[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:38:27 dist-test-slave-lnj3 krb5kdc[5982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:38:27 dist-test-slave-lnj3 krb5kdc[5982](info): set up 2 sockets
Jun 18 22:38:27 dist-test-slave-lnj3 krb5kdc[5982](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260618 22:38:28.875730 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.002s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:38:28 dist-test-slave-lnj3 krb5kdc[5982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:38:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:38:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:38:29.026602 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:44037
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:44763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:44037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:29.325152 5995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:29.325459 5995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:29.325608 5995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:29.332767 5995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:38:29.332885 5995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:29.332962 5995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:29.333034 5995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:38:29.333092 5995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:38:29.342900 5995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:44763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:44037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:44037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:29.344190 5995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:29.345453 5995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:29.356959 6000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:29.356871 6001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:29.356894 6003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:29.357501 5995 server_base.cc:1034] running on GCE node
I20260618 22:38:29.358196 5995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:29.359527 5995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:29.360816 5995 hybrid_clock.cc:648] HybridClock initialized: now 1781822309360754 us; error 41 us; skew 500 ppm
I20260618 22:38:29.361160 5995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:29 dist-test-slave-lnj3 krb5kdc[5982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822309, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:29.364044 5995 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:38:29.365375 5995 webserver.cc:460] Webserver started at http://127.5.213.254:44599/ using document root <none> and password file <none>
I20260618 22:38:29.365996 5995 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:29.366091 5995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:29.366320 5995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:29.369264 5995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "9bb4b2dd615d479cb8b563864967f8c0"
format_stamp: "Formatted at 2026-06-18 22:38:29 on dist-test-slave-lnj3"
server_key: "8ad12e4b48d957000dc94bf434324dc4"
server_key_iv: "b263afab7f1fb6d17aa44aebbb981826"
server_key_version: "encryptionkey@0"
I20260618 22:38:29.369944 5995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "9bb4b2dd615d479cb8b563864967f8c0"
format_stamp: "Formatted at 2026-06-18 22:38:29 on dist-test-slave-lnj3"
server_key: "8ad12e4b48d957000dc94bf434324dc4"
server_key_iv: "b263afab7f1fb6d17aa44aebbb981826"
server_key_version: "encryptionkey@0"
I20260618 22:38:29.374869 5995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260618 22:38:29.378078 6010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:29.379350 5995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:38:29.379503 5995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "9bb4b2dd615d479cb8b563864967f8c0"
format_stamp: "Formatted at 2026-06-18 22:38:29 on dist-test-slave-lnj3"
server_key: "8ad12e4b48d957000dc94bf434324dc4"
server_key_iv: "b263afab7f1fb6d17aa44aebbb981826"
server_key_version: "encryptionkey@0"
I20260618 22:38:29.379706 5995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:29.403425 5995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:29.407402 5995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:29.407629 5995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:29.423560 5995 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:44037
I20260618 22:38:29.423570 6061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:44037 every 8 connection(s)
I20260618 22:38:29.424846 5995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:38:29.428632 6062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:29.430011 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 5995
I20260618 22:38:29.430284 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:38:29.432107 5975 external_mini_cluster.cc:1428] Setting key a0fb046162f37d2a27e361de1e1867ee
I20260618 22:38:29.439687 6062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0: Bootstrap starting.
I20260618 22:38:29.442566 6062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:29.443542 6062 log.cc:824] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0: Log is configured to *not* fsync() on all Append() calls
I20260618 22:38:29.446025 6062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0: No bootstrap required, opened a new log
I20260618 22:38:29.455753 6062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb4b2dd615d479cb8b563864967f8c0" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44037 } }
I20260618 22:38:29.456076 6062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:29.456207 6062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bb4b2dd615d479cb8b563864967f8c0, State: Initialized, Role: FOLLOWER
I20260618 22:38:29.456784 6062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb4b2dd615d479cb8b563864967f8c0" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44037 } }
I20260618 22:38:29.457031 6062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:38:29.457172 6062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:38:29.457360 6062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:29.460347 6062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb4b2dd615d479cb8b563864967f8c0" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44037 } }
I20260618 22:38:29.460949 6062 leader_election.cc:302] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bb4b2dd615d479cb8b563864967f8c0; no voters:
I20260618 22:38:29.461603 6062 leader_election.cc:288] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:38:29.461706 6067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0 [term 1 FOLLOWER]: Leader election won for term 1
Jun 18 22:38:29 dist-test-slave-lnj3 krb5kdc[5982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:29.462788 6067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0 [term 1 LEADER]: Becoming Leader. State: Replica: 9bb4b2dd615d479cb8b563864967f8c0, State: Running, Role: LEADER
I20260618 22:38:29.463302 6067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb4b2dd615d479cb8b563864967f8c0" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44037 } }
I20260618 22:38:29.463867 6062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:38:29.466195 6068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bb4b2dd615d479cb8b563864967f8c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb4b2dd615d479cb8b563864967f8c0" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44037 } } }
I20260618 22:38:29.466929 6068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0 [sys.catalog]: This master's current role is: LEADER
I20260618 22:38:29.465935 6069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bb4b2dd615d479cb8b563864967f8c0. Latest consensus state: current_term: 1 leader_uuid: "9bb4b2dd615d479cb8b563864967f8c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb4b2dd615d479cb8b563864967f8c0" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44037 } } }
I20260618 22:38:29.469203 6069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0 [sys.catalog]: This master's current role is: LEADER
I20260618 22:38:29.470935 6073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:38:29.475172 6064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:29.435691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56510 (local address 127.5.213.254:44037)
0618 22:38:29.438889 (+ 3198us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:29.438905 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:29.440181 (+ 1276us) server_negotiation.cc:407] Connection header received
0618 22:38:29.448439 (+ 8258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:29.448466 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:29.448947 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:29.449378 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:29.452061 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:29.454412 (+ 2351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:29.456185 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:29.456960 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:29.457144 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:29.464236 (+ 7092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:29.464247 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:29.464267 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:29.464316 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:29.468537 (+ 4221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:29.469708 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:29.469719 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:29.469727 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:29.469990 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:29.470595 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:29.470600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:29.470603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:29.471197 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:29.472991 (+ 1794us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:29.473297 (+ 306us) server_negotiation.cc:299] Negotiation successful
0618 22:38:29.473726 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2963,"thread_start_us":2709,"threads_started":1}
I20260618 22:38:29.477385 6073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:38:29.486660 6073 catalog_manager.cc:1347] Generated new cluster ID: d6fd4c32bbd04e0386d2ee2f2cdc099f
I20260618 22:38:29.486778 6073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:38:29.499703 6073 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:38:29.501304 6073 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:38:29.508976 6073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bb4b2dd615d479cb8b563864967f8c0: Generated new TSK 0
I20260618 22:38:29.509905 6073 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
I20260618 22:38:29.552399 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44037
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:44763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:29.851892 6088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:29.852140 6088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:29.852254 6088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:29.859105 6088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:29.859200 6088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:29.859347 6088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:38:29.868551 6088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:44763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44037
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:29.869735 6088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:29.871018 6088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:29.883643 6093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:29.883563 6096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:29.883622 6094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:29.884729 6088 server_base.cc:1034] running on GCE node
I20260618 22:38:29.885330 6088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:29.886250 6088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:29.887459 6088 hybrid_clock.cc:648] HybridClock initialized: now 1781822309887406 us; error 32 us; skew 500 ppm
I20260618 22:38:29.887763 6088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:29 dist-test-slave-lnj3 krb5kdc[5982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822309, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:29.890619 6088 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:38:29.891907 6088 webserver.cc:460] Webserver started at http://127.5.213.193:34751/ using document root <none> and password file <none>
I20260618 22:38:29.892527 6088 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:29.892639 6088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:29.892912 6088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:29.895609 6088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "676038174a364e7994a684c647c06833"
format_stamp: "Formatted at 2026-06-18 22:38:29 on dist-test-slave-lnj3"
server_key: "a90a9bf3cb8b8be3a93d432ead140ccb"
server_key_iv: "654bb03328cfee67b9a555865d23267b"
server_key_version: "encryptionkey@0"
I20260618 22:38:29.896318 6088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "676038174a364e7994a684c647c06833"
format_stamp: "Formatted at 2026-06-18 22:38:29 on dist-test-slave-lnj3"
server_key: "a90a9bf3cb8b8be3a93d432ead140ccb"
server_key_iv: "654bb03328cfee67b9a555865d23267b"
server_key_version: "encryptionkey@0"
I20260618 22:38:29.900835 6088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260618 22:38:29.903882 6103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:29.905028 6088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:38:29.905164 6088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "676038174a364e7994a684c647c06833"
format_stamp: "Formatted at 2026-06-18 22:38:29 on dist-test-slave-lnj3"
server_key: "a90a9bf3cb8b8be3a93d432ead140ccb"
server_key_iv: "654bb03328cfee67b9a555865d23267b"
server_key_version: "encryptionkey@0"
I20260618 22:38:29.905319 6088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:29.926002 6088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:29.929807 6088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:29.930038 6088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:29.930907 6088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:38:29.932114 6088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:29.932197 6088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:29.932309 6088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:29.932359 6088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:29.959721 6088 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:40733
I20260618 22:38:29.959772 6215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:40733 every 8 connection(s)
I20260618 22:38:29.960919 6088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:38:29.966215 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 6088
I20260618 22:38:29.966428 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:38:29.967213 5975 external_mini_cluster.cc:1428] Setting key 8320b1d9e1a1a1c983176904873e26e1
Jun 18 22:38:29 dist-test-slave-lnj3 krb5kdc[5982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822309, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:29.981597 6064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:29.963263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:57051 (local address 127.5.213.254:44037)
0618 22:38:29.963426 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:29.963430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:29.964568 (+ 1138us) server_negotiation.cc:407] Connection header received
0618 22:38:29.966591 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:29.966595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:29.966698 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:29.966780 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:29.968590 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:29.969577 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:29.971117 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:29.971788 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:29.971958 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:29.975323 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:29.975328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:29.975332 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:29.975365 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:29.977837 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:29.978731 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:29.978735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:29.978737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:29.978859 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:29.979372 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:29.979377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:29.979379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:29.979751 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:29.979888 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:29.980792 (+ 904us) server_negotiation.cc:299] Negotiation successful
0618 22:38:29.981140 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.5.213.194@KRBTEST.COM; defaulting to no policy
I20260618 22:38:29.982610 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:29.963743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:44037 (local address 127.5.213.193:57051)
0618 22:38:29.964419 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:29.964462 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:29.966305 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:29.967046 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:29.967058 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:29.967750 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:29.968390 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:29.968413 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:29.969737 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:29.969741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:29.970984 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:29.970995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:29.972017 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:29.972021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:29.972218 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:29.972902 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:29.972927 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:29.975086 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:29.977967 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:29.977976 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:29.977993 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:29.978565 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:29.978970 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:29.978975 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:29.978979 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:29.979229 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:29.979887 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:29.979920 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:29.980327 (+ 407us) client_negotiation.cc:769] Sending connection context
0618 22:38:29.980643 (+ 316us) client_negotiation.cc:240] Negotiation successful
0618 22:38:29.981177 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":245,"threads_started":1}
I20260618 22:38:29.984380 6216 heartbeater.cc:347] Connected to a master server at 127.5.213.254:44037
I20260618 22:38:29.984707 6216 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.194@KRBTEST.COM" created.
I20260618 22:38:29.985553 6216 heartbeater.cc:511] Master 127.5.213.254:44037 requested a full tablet report, sending...
I20260618 22:38:29.987839 6027 ts_manager.cc:194] Registered new tserver with Master: 676038174a364e7994a684c647c06833 (127.5.213.193:40733)
I20260618 22:38:29.989605 6027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:57051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
I20260618 22:38:30.005637 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44037
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:44763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:30.303530 6221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:30.303808 6221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:30.303942 6221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:30.311167 6221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:30.311266 6221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:30.311425 6221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:38:30.320713 6221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:44763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44037
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:30.321923 6221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:30.323184 6221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:30.334905 6226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:30.334931 6227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:30.334968 6229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:30.335945 6221 server_base.cc:1034] running on GCE node
I20260618 22:38:30.336483 6221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:30.337322 6221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:30.338581 6221 hybrid_clock.cc:648] HybridClock initialized: now 1781822310338492 us; error 32 us; skew 500 ppm
I20260618 22:38:30.338922 6221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:30 dist-test-slave-lnj3 krb5kdc[5982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822310, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:30.341984 6221 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:38:30.343428 6221 webserver.cc:460] Webserver started at http://127.5.213.194:34505/ using document root <none> and password file <none>
I20260618 22:38:30.344079 6221 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:30.344210 6221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:30.344521 6221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:30.347456 6221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "735bc14468714fbdb031b2b9a6b751d2"
format_stamp: "Formatted at 2026-06-18 22:38:30 on dist-test-slave-lnj3"
server_key: "3cfb200c5992a268f82f77085ae0eab3"
server_key_iv: "1a301e153241e5a420fc80bf2d3f6bb5"
server_key_version: "encryptionkey@0"
I20260618 22:38:30.348265 6221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "735bc14468714fbdb031b2b9a6b751d2"
format_stamp: "Formatted at 2026-06-18 22:38:30 on dist-test-slave-lnj3"
server_key: "3cfb200c5992a268f82f77085ae0eab3"
server_key_iv: "1a301e153241e5a420fc80bf2d3f6bb5"
server_key_version: "encryptionkey@0"
I20260618 22:38:30.353438 6221 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260618 22:38:30.356745 6236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:30.358047 6221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:38:30.358196 6221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "735bc14468714fbdb031b2b9a6b751d2"
format_stamp: "Formatted at 2026-06-18 22:38:30 on dist-test-slave-lnj3"
server_key: "3cfb200c5992a268f82f77085ae0eab3"
server_key_iv: "1a301e153241e5a420fc80bf2d3f6bb5"
server_key_version: "encryptionkey@0"
I20260618 22:38:30.358361 6221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:30.372081 6221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:30.376029 6221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:30.376284 6221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:30.377152 6221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:38:30.378369 6221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:30.378440 6221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:30.378556 6221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:30.378607 6221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:30.406418 6221 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:44651
I20260618 22:38:30.406461 6348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:44651 every 8 connection(s)
I20260618 22:38:30.407666 6221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:38:30.409160 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 6221
I20260618 22:38:30.409329 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:38:30.409718 5975 external_mini_cluster.cc:1428] Setting key 16d10a2673b88842d2055d2270cac099
Jun 18 22:38:30 dist-test-slave-lnj3 krb5kdc[5982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822310, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.195@KRBTEST.COM" created.
I20260618 22:38:30.428026 6064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:30.410262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:39217 (local address 127.5.213.254:44037)
0618 22:38:30.410454 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:30.410460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:30.411883 (+ 1423us) server_negotiation.cc:407] Connection header received
0618 22:38:30.413634 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:30.413639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:30.413713 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:30.413792 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:30.415405 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:30.416124 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:30.418292 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:30.418800 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:30.418924 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:30.422112 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:30.422118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:30.422123 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:30.422153 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:30.424812 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:30.425517 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:30.425522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:30.425525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:30.425657 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:30.426168 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:30.426174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:30.426178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:30.426519 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:30.426720 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:30.427471 (+ 751us) server_negotiation.cc:299] Negotiation successful
0618 22:38:30.427699 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 22:38:30.429171 6351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:30.410861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:44037 (local address 127.5.213.194:39217)
0618 22:38:30.411735 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:30.411774 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:30.413388 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:30.414031 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:30.414040 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:30.414512 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:30.415234 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:30.415259 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:30.416259 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:30.416274 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:30.418151 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:30.418165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:30.419029 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:30.419036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:30.419271 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:30.419866 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:30.419888 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:30.421921 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:30.424945 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:30.424953 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:30.424969 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:30.425407 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:30.425756 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:30.425761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:30.425765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:30.426006 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:30.426680 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:30.426688 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:30.427083 (+ 395us) client_negotiation.cc:769] Sending connection context
0618 22:38:30.427378 (+ 295us) client_negotiation.cc:240] Negotiation successful
0618 22:38:30.427885 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":324,"threads_started":1}
I20260618 22:38:30.430593 6349 heartbeater.cc:347] Connected to a master server at 127.5.213.254:44037
I20260618 22:38:30.430943 6349 heartbeater.cc:464] Registering TS with master...
I20260618 22:38:30.431630 6349 heartbeater.cc:511] Master 127.5.213.254:44037 requested a full tablet report, sending...
I20260618 22:38:30.432916 6027 ts_manager.cc:194] Registered new tserver with Master: 735bc14468714fbdb031b2b9a6b751d2 (127.5.213.194:44651)
I20260618 22:38:30.433696 6027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:39217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
I20260618 22:38:30.447974 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44037
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:44763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:30.745541 6354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:30.745810 6354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:30.745909 6354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:30.752818 6354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:30.752911 6354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:30.753028 6354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:38:30.762207 6354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:44763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44037
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:30.763401 6354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:30.764688 6354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:30.776727 6362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:30.776813 6359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:30.776787 6360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:30.777329 6354 server_base.cc:1034] running on GCE node
I20260618 22:38:30.777889 6354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:30.778777 6354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:30.779960 6354 hybrid_clock.cc:648] HybridClock initialized: now 1781822310779913 us; error 50 us; skew 500 ppm
I20260618 22:38:30.780237 6354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:30 dist-test-slave-lnj3 krb5kdc[5982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822310, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:30.783221 6354 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:38:30.784571 6354 webserver.cc:460] Webserver started at http://127.5.213.195:42441/ using document root <none> and password file <none>
I20260618 22:38:30.785190 6354 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:30.785297 6354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:30.785574 6354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:30.788331 6354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "d984a55a91cb40789e39c766e335bae9"
format_stamp: "Formatted at 2026-06-18 22:38:30 on dist-test-slave-lnj3"
server_key: "4234e8c063b2a9785b2e4d1db055d3d5"
server_key_iv: "aa5e7468da437e8a712c574e1b2f832f"
server_key_version: "encryptionkey@0"
I20260618 22:38:30.789014 6354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "d984a55a91cb40789e39c766e335bae9"
format_stamp: "Formatted at 2026-06-18 22:38:30 on dist-test-slave-lnj3"
server_key: "4234e8c063b2a9785b2e4d1db055d3d5"
server_key_iv: "aa5e7468da437e8a712c574e1b2f832f"
server_key_version: "encryptionkey@0"
I20260618 22:38:30.793823 6354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260618 22:38:30.797255 6369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:30.798558 6354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260618 22:38:30.798770 6354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "d984a55a91cb40789e39c766e335bae9"
format_stamp: "Formatted at 2026-06-18 22:38:30 on dist-test-slave-lnj3"
server_key: "4234e8c063b2a9785b2e4d1db055d3d5"
server_key_iv: "aa5e7468da437e8a712c574e1b2f832f"
server_key_version: "encryptionkey@0"
I20260618 22:38:30.798957 6354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:30.811672 6354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:30.815548 6354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:30.815801 6354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:30.816689 6354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:38:30.818022 6354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:30.818106 6354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:30.818233 6354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:30.818288 6354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:30.847441 6354 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:41651
I20260618 22:38:30.847488 6481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:41651 every 8 connection(s)
I20260618 22:38:30.848680 6354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:38:30.850901 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 6354
I20260618 22:38:30.851087 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:38:30.851608 5975 external_mini_cluster.cc:1428] Setting key 681ec2ea49988352710467379a7ff9ff
Jun 18 22:38:30 dist-test-slave-lnj3 krb5kdc[5982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822310, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:30.868287 6064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:30.851159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:33187 (local address 127.5.213.254:44037)
0618 22:38:30.851302 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:30.851306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:30.852778 (+ 1472us) server_negotiation.cc:407] Connection header received
0618 22:38:30.855005 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:30.855010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:30.855087 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:30.855169 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:30.856614 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:30.857290 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:30.858768 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:30.859188 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:30.859343 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:30.863015 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:30.863021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:30.863024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:30.863055 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:30.865446 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:30.866169 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:30.866174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:30.866177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:30.866322 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:30.866817 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:30.866822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:30.866825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:30.867127 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:30.867273 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:30.867907 (+ 634us) server_negotiation.cc:299] Negotiation successful
0618 22:38:30.868071 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260618 22:38:30.869165 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:30.851838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:44037 (local address 127.5.213.195:33187)
0618 22:38:30.852618 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:30.852674 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:30.854734 (+ 2060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:30.855402 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:30.855410 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:30.855897 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:30.856459 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:30.856476 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:30.857427 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:30.857432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:30.858622 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:30.858662 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:30.859417 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:30.859425 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:30.859646 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:30.860211 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:30.860239 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:30.862782 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:30.865580 (+ 2798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:30.865589 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:30.865604 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:30.866037 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:30.866435 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:30.866440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:30.866443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:30.866720 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:30.867254 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:30.867261 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:30.867599 (+ 338us) client_negotiation.cc:769] Sending connection context
0618 22:38:30.867814 (+ 215us) client_negotiation.cc:240] Negotiation successful
0618 22:38:30.868163 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":288,"threads_started":1}
I20260618 22:38:30.870314 6482 heartbeater.cc:347] Connected to a master server at 127.5.213.254:44037
I20260618 22:38:30.870611 6482 heartbeater.cc:464] Registering TS with master...
I20260618 22:38:30.871353 6482 heartbeater.cc:511] Master 127.5.213.254:44037 requested a full tablet report, sending...
I20260618 22:38:30.872649 6027 ts_manager.cc:194] Registered new tserver with Master: d984a55a91cb40789e39c766e335bae9 (127.5.213.195:41651)
I20260618 22:38:30.873369 6027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:33187
I20260618 22:38:30.879235 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 22:38:30 dist-test-slave-lnj3 krb5kdc[5982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 22:38:30 dist-test-slave-lnj3 krb5kdc[5982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:38:30.918289 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:30.904052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.5.213.193:40733)
0618 22:38:30.904838 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:30.904846 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:30.904872 (+ 26us) server_negotiation.cc:407] Connection header received
0618 22:38:30.904929 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:30.904934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:30.905136 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:30.905262 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:30.906518 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:30.907549 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:30.908872 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:30.909429 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:30.909523 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:30.913278 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:30.913289 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:30.913303 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:30.913341 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:30.915943 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:30.916510 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:30.916517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:30.916519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:30.916636 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:30.916991 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:30.916994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:30.916996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:30.917298 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:30.917510 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:30.917755 (+ 245us) server_negotiation.cc:299] Negotiation successful
0618 22:38:30.917957 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":567,"threads_started":1}
W20260618 22:38:30.919510 6131 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39310
Jun 18 22:38:30 dist-test-slave-lnj3 krb5kdc[5982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 22:38:30 dist-test-slave-lnj3 krb5kdc[5982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:38:30.965749 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:30.946923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39316 (local address 127.5.213.193:40733)
0618 22:38:30.947126 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:30.947133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:30.947294 (+ 161us) server_negotiation.cc:407] Connection header received
0618 22:38:30.947593 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:30.947599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:30.947684 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:30.947789 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:30.949458 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:30.950279 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:30.951873 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:30.952282 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:30.952387 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:30.959976 (+ 7589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:30.959982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:30.959985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:30.960021 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:30.962980 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:30.963746 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:30.963750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:30.963753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:30.963888 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:30.964264 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:30.964269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:30.964271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:30.964543 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:30.964725 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:30.965222 (+ 497us) server_negotiation.cc:299] Negotiation successful
0618 22:38:30.965431 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:38:30.967195 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 6088
I20260618 22:38:30.978546 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:30.968207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39322 (local address 127.5.213.193:40733)
0618 22:38:30.968387 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:30.968392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:30.968601 (+ 209us) server_negotiation.cc:407] Connection header received
0618 22:38:30.968695 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:30.968699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:30.968754 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:30.968824 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:30.969813 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:30.970473 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:30.971998 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:30.972393 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:30.972479 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:30.974379 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:30.974383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:30.974387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:30.974413 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:30.976616 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:30.977150 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:30.977153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:30.977156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:30.977272 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:30.977510 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:30.977512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:30.977514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:30.977740 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:30.977892 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:30.978079 (+ 187us) server_negotiation.cc:299] Negotiation successful
0618 22:38:30.978234 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 22:38:30.979243 6211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:38:31.058203 6216 heartbeater.cc:503] Master 127.5.213.254:44037 was elected leader, sending a full tablet report...
I20260618 22:38:31.058866 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 6088
W20260618 22:38:31.075948 6011 connection.cc:535] server connection from 127.5.213.193:57051 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260618 22:38:31.076254 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 6221
Jun 18 22:38:31 dist-test-slave-lnj3 krb5kdc[5982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.194@KRBTEST.COM
I20260618 22:38:31.090600 6496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:31.077312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49228 (local address 127.5.213.194:44651)
0618 22:38:31.077840 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:31.077848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:31.077873 (+ 25us) server_negotiation.cc:407] Connection header received
0618 22:38:31.077928 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:31.077935 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:31.078140 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:31.078262 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:31.079369 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:31.080339 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:31.081475 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:31.082013 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:31.082106 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:31.085855 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:31.085866 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:31.085881 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:31.085913 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:31.088430 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:31.088986 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:31.088992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:31.088995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:31.089103 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:31.089378 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:31.089381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:31.089383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:31.089686 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:31.089900 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:31.090147 (+ 247us) server_negotiation.cc:299] Negotiation successful
0618 22:38:31.090327 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":261,"threads_started":1}
I20260618 22:38:31.091957 6344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:38:31.170284 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 6221
I20260618 22:38:31.183343 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 6354
Jun 18 22:38:31 dist-test-slave-lnj3 krb5kdc[5982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.195@KRBTEST.COM
I20260618 22:38:31.198593 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:31.184314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43330 (local address 127.5.213.195:41651)
0618 22:38:31.184904 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:31.184911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:31.184938 (+ 27us) server_negotiation.cc:407] Connection header received
0618 22:38:31.184993 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:31.184999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:31.185201 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:31.185339 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:31.186524 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:31.187552 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:31.188797 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:31.189322 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:31.189429 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:31.193226 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:31.193237 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:31.193250 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:31.193281 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:31.196052 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:31.196680 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:31.196689 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:31.196692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:31.196809 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:31.197130 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:31.197135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:31.197139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:31.197463 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:31.197764 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:31.198026 (+ 262us) server_negotiation.cc:299] Negotiation successful
0618 22:38:31.198267 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":299,"threads_started":1}
I20260618 22:38:31.199837 6477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:38:31.279076 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 6354
I20260618 22:38:31.290968 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 5995
I20260618 22:38:31.292074 6057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:38:31.389402 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 5995
2026-06-18T22:38:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3520 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:38:31 dist-test-slave-lnj3 krb5kdc[6503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:38:31 dist-test-slave-lnj3 krb5kdc[6503](info): set up 2 sockets
Jun 18 22:38:31 dist-test-slave-lnj3 krb5kdc[6503](info): commencing operation
krb5kdc: starting...
W20260618 22:38:32.386237 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.002s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:38:32 dist-test-slave-lnj3 krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:38:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:38:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:38:32.543839 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:44623
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:44623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:32.842378 6516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:32.842700 6516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:32.842830 6516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:32.849978 6516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:38:32.850073 6516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:32.850140 6516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:32.850214 6516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:38:32.850267 6516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:38:32.859907 6516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:44623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:44623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:32.861109 6516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:32.862334 6516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:32.872888 6521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:32.873041 6524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:32.872907 6522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:32.873692 6516 server_base.cc:1034] running on GCE node
I20260618 22:38:32.874338 6516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:32.875658 6516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:32.876863 6516 hybrid_clock.cc:648] HybridClock initialized: now 1781822312876814 us; error 40 us; skew 500 ppm
I20260618 22:38:32.877192 6516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:32 dist-test-slave-lnj3 krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822312, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:32.880092 6516 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:38:32.881472 6516 webserver.cc:460] Webserver started at http://127.5.213.254:33365/ using document root <none> and password file <none>
I20260618 22:38:32.882098 6516 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:32.882190 6516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:32.882434 6516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:32.885421 6516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "d4f3205f02cc4fb5b065c4e871c187f3"
format_stamp: "Formatted at 2026-06-18 22:38:32 on dist-test-slave-lnj3"
server_key: "53a82052a35ed707c777bf9ea30b4e78"
server_key_iv: "a495bf1e904d42ef4d40a249e9f04957"
server_key_version: "encryptionkey@0"
I20260618 22:38:32.886076 6516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "d4f3205f02cc4fb5b065c4e871c187f3"
format_stamp: "Formatted at 2026-06-18 22:38:32 on dist-test-slave-lnj3"
server_key: "53a82052a35ed707c777bf9ea30b4e78"
server_key_iv: "a495bf1e904d42ef4d40a249e9f04957"
server_key_version: "encryptionkey@0"
I20260618 22:38:32.890974 6516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260618 22:38:32.894152 6531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:32.895443 6516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:38:32.895573 6516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "d4f3205f02cc4fb5b065c4e871c187f3"
format_stamp: "Formatted at 2026-06-18 22:38:32 on dist-test-slave-lnj3"
server_key: "53a82052a35ed707c777bf9ea30b4e78"
server_key_iv: "a495bf1e904d42ef4d40a249e9f04957"
server_key_version: "encryptionkey@0"
I20260618 22:38:32.895752 6516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:32.908465 6516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:32.912508 6516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:32.912732 6516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:32.929355 6516 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:44623
I20260618 22:38:32.929387 6582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:44623 every 8 connection(s)
I20260618 22:38:32.930718 6516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:38:32.934404 6583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:32.937933 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 6516
I20260618 22:38:32.938155 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:38:32.938964 5975 external_mini_cluster.cc:1428] Setting key 79820a788974fd2ded5d95b489216452
I20260618 22:38:32.947084 6583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3: Bootstrap starting.
I20260618 22:38:32.950642 6583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:32.951869 6583 log.cc:824] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3: Log is configured to *not* fsync() on all Append() calls
Jun 18 22:38:32 dist-test-slave-lnj3 krb5kdc[6503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:32.954941 6583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3: No bootstrap required, opened a new log
I20260618 22:38:32.960968 6586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:32.940897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52080 (local address 127.5.213.254:44623)
0618 22:38:32.941584 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:32.941594 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:32.941633 (+ 39us) server_negotiation.cc:407] Connection header received
0618 22:38:32.943218 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:32.943240 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:32.943631 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:32.943949 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:32.945544 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:32.946945 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:32.948587 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:32.949243 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:32.949366 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:32.954019 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:32.954026 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:32.954046 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:32.954077 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:32.956968 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:32.957741 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:32.957747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:32.957754 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:32.957913 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:32.958261 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:32.958264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:32.958266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:32.958831 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:32.959031 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:32.959563 (+ 532us) server_negotiation.cc:299] Negotiation successful
0618 22:38:32.959951 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":321,"threads_started":1}
I20260618 22:38:32.963789 6583 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f3205f02cc4fb5b065c4e871c187f3" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44623 } }
I20260618 22:38:32.964208 6583 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:32.964299 6583 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4f3205f02cc4fb5b065c4e871c187f3, State: Initialized, Role: FOLLOWER
I20260618 22:38:32.964834 6583 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f3205f02cc4fb5b065c4e871c187f3" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44623 } }
I20260618 22:38:32.965041 6583 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:38:32.965170 6583 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:38:32.965319 6583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:32.967600 6583 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f3205f02cc4fb5b065c4e871c187f3" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44623 } }
I20260618 22:38:32.968053 6583 leader_election.cc:302] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4f3205f02cc4fb5b065c4e871c187f3; no voters:
I20260618 22:38:32.968569 6583 leader_election.cc:288] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:38:32.968670 6588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:38:32.969242 6588 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3 [term 1 LEADER]: Becoming Leader. State: Replica: d4f3205f02cc4fb5b065c4e871c187f3, State: Running, Role: LEADER
I20260618 22:38:32.969712 6588 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f3205f02cc4fb5b065c4e871c187f3" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44623 } }
I20260618 22:38:32.970403 6583 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:38:32.972134 6589 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4f3205f02cc4fb5b065c4e871c187f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f3205f02cc4fb5b065c4e871c187f3" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44623 } } }
I20260618 22:38:32.972500 6589 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3 [sys.catalog]: This master's current role is: LEADER
I20260618 22:38:32.972955 6590 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4f3205f02cc4fb5b065c4e871c187f3. Latest consensus state: current_term: 1 leader_uuid: "d4f3205f02cc4fb5b065c4e871c187f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f3205f02cc4fb5b065c4e871c187f3" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44623 } } }
I20260618 22:38:32.973212 6590 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3 [sys.catalog]: This master's current role is: LEADER
I20260618 22:38:32.974048 6593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:38:32.977829 6593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:38:32.987457 6593 catalog_manager.cc:1347] Generated new cluster ID: 4e3af167ea514f6985e6b71b08c0d7f0
I20260618 22:38:32.987555 6593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:38:33.012662 6593 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:38:33.014008 6593 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:38:33.031988 6593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4f3205f02cc4fb5b065c4e871c187f3: Generated new TSK 0
I20260618 22:38:33.033013 6593 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
I20260618 22:38:33.076187 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44623
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:33.383141 6609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:33.383432 6609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:33.383569 6609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:33.390854 6609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:33.390967 6609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:33.391112 6609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:38:33.401278 6609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44623
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:33.402565 6609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:33.403900 6609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:33.416523 6614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:33.416527 6617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:33.416973 6615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:33.417207 6609 server_base.cc:1034] running on GCE node
I20260618 22:38:33.417789 6609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:33.418751 6609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:33.419976 6609 hybrid_clock.cc:648] HybridClock initialized: now 1781822313419911 us; error 48 us; skew 500 ppm
I20260618 22:38:33.420296 6609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:33 dist-test-slave-lnj3 krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822313, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:33.423403 6609 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:38:33.424894 6609 webserver.cc:460] Webserver started at http://127.5.213.193:43101/ using document root <none> and password file <none>
I20260618 22:38:33.425559 6609 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:33.425665 6609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:33.425957 6609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:33.429136 6609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "e69069ebd26e4cb1a4dd83099d3214bb"
format_stamp: "Formatted at 2026-06-18 22:38:33 on dist-test-slave-lnj3"
server_key: "51a8a963ffd10c1c56a77effdf0e3e1f"
server_key_iv: "fc18a1c0a4dc143a11b82a1590a10362"
server_key_version: "encryptionkey@0"
I20260618 22:38:33.429929 6609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "e69069ebd26e4cb1a4dd83099d3214bb"
format_stamp: "Formatted at 2026-06-18 22:38:33 on dist-test-slave-lnj3"
server_key: "51a8a963ffd10c1c56a77effdf0e3e1f"
server_key_iv: "fc18a1c0a4dc143a11b82a1590a10362"
server_key_version: "encryptionkey@0"
I20260618 22:38:33.435025 6609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260618 22:38:33.438366 6624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:33.439714 6609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:38:33.439864 6609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "e69069ebd26e4cb1a4dd83099d3214bb"
format_stamp: "Formatted at 2026-06-18 22:38:33 on dist-test-slave-lnj3"
server_key: "51a8a963ffd10c1c56a77effdf0e3e1f"
server_key_iv: "fc18a1c0a4dc143a11b82a1590a10362"
server_key_version: "encryptionkey@0"
I20260618 22:38:33.440047 6609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:33.465976 6609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:33.469995 6609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:33.470263 6609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:33.471279 6609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:38:33.472666 6609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:33.472745 6609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:33.472843 6609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:33.472896 6609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:33.502353 6609 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:37953
I20260618 22:38:33.502398 6736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:37953 every 8 connection(s)
I20260618 22:38:33.503728 6609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:38:33.511868 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 6609
I20260618 22:38:33.512068 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:38:33.512809 5975 external_mini_cluster.cc:1428] Setting key 7b828349d5fb26367c8d54d5f5241435
Jun 18 22:38:33 dist-test-slave-lnj3 krb5kdc[6503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822313, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:33.523861 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:33.506129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:42443 (local address 127.5.213.254:44623)
0618 22:38:33.506548 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:33.506554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:33.507652 (+ 1098us) server_negotiation.cc:407] Connection header received
0618 22:38:33.509412 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:33.509416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:33.509493 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:33.509588 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:33.511191 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:33.511896 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:33.514031 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:33.514466 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:33.514575 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:33.518145 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:33.518149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:33.518153 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:33.518186 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:33.520672 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:33.521465 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:33.521470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:33.521472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:33.521602 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:33.522016 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:33.522020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:33.522022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:33.522303 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:33.522432 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:33.523228 (+ 796us) server_negotiation.cc:299] Negotiation successful
0618 22:38:33.523473 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":186,"threads_started":1}
I20260618 22:38:33.524892 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:33.506787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:44623 (local address 127.5.213.193:42443)
0618 22:38:33.507485 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:33.507524 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:33.509133 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:33.509796 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:33.509804 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:33.510350 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:33.511000 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:33.511018 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:33.512073 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:33.512079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:33.513903 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:33.513914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:33.514707 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:33.514716 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:33.514944 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:33.515603 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:33.515625 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:33.517916 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:33.520834 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:33.520845 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:33.520866 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:33.521332 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:33.521707 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:33.521713 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:33.521716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:33.521903 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:33.522432 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:33.522441 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:33.522855 (+ 414us) client_negotiation.cc:769] Sending connection context
0618 22:38:33.523124 (+ 269us) client_negotiation.cc:240] Negotiation successful
0618 22:38:33.523732 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":246,"threads_started":1}
I20260618 22:38:33.526736 6737 heartbeater.cc:347] Connected to a master server at 127.5.213.254:44623
I20260618 22:38:33.527040 6737 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.213.194@KRBTEST.COM; defaulting to no policy
I20260618 22:38:33.527822 6737 heartbeater.cc:511] Master 127.5.213.254:44623 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.194@KRBTEST.COM" created.
I20260618 22:38:33.530292 6548 ts_manager.cc:194] Registered new tserver with Master: e69069ebd26e4cb1a4dd83099d3214bb (127.5.213.193:37953)
I20260618 22:38:33.532325 6548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:42443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
I20260618 22:38:33.550499 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44623
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:33.857787 6743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:33.858072 6743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:33.858196 6743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:33.865286 6743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:33.865389 6743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:33.865535 6743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:38:33.875133 6743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44623
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:33.876418 6743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:33.877724 6743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:33.890074 6748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:33.890139 6751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:33.890154 6749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:33.890875 6743 server_base.cc:1034] running on GCE node
I20260618 22:38:33.891505 6743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:33.892431 6743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:33.893621 6743 hybrid_clock.cc:648] HybridClock initialized: now 1781822313893568 us; error 44 us; skew 500 ppm
I20260618 22:38:33.893936 6743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:33 dist-test-slave-lnj3 krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822313, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:33.896956 6743 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:38:33.898411 6743 webserver.cc:460] Webserver started at http://127.5.213.194:37503/ using document root <none> and password file <none>
I20260618 22:38:33.899127 6743 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:33.899223 6743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:33.899492 6743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:33.902235 6743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "ca6dabf36e3249208ea8a3b4f2fae3c2"
format_stamp: "Formatted at 2026-06-18 22:38:33 on dist-test-slave-lnj3"
server_key: "89f426ebd8875d6c823243208322a1d2"
server_key_iv: "31bd78dcece4df4416ce19a7a23ab090"
server_key_version: "encryptionkey@0"
I20260618 22:38:33.902987 6743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "ca6dabf36e3249208ea8a3b4f2fae3c2"
format_stamp: "Formatted at 2026-06-18 22:38:33 on dist-test-slave-lnj3"
server_key: "89f426ebd8875d6c823243208322a1d2"
server_key_iv: "31bd78dcece4df4416ce19a7a23ab090"
server_key_version: "encryptionkey@0"
I20260618 22:38:33.907883 6743 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260618 22:38:33.911154 6758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:33.912431 6743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:38:33.912570 6743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "ca6dabf36e3249208ea8a3b4f2fae3c2"
format_stamp: "Formatted at 2026-06-18 22:38:33 on dist-test-slave-lnj3"
server_key: "89f426ebd8875d6c823243208322a1d2"
server_key_iv: "31bd78dcece4df4416ce19a7a23ab090"
server_key_version: "encryptionkey@0"
I20260618 22:38:33.912743 6743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:33.931999 6743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:33.936132 6743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:33.936429 6743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:33.937410 6743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:38:33.938807 6743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:33.938896 6743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:33.939008 6743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:33.939060 6743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:33.968314 6743 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:38685
I20260618 22:38:33.968361 6870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:38685 every 8 connection(s)
I20260618 22:38:33.969571 6743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:38:33.975477 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 6743
I20260618 22:38:33.975659 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:38:33.976037 5975 external_mini_cluster.cc:1428] Setting key a3de0cc1f2ad7746a818690aa9088bf8
Jun 18 22:38:33 dist-test-slave-lnj3 krb5kdc[6503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822313, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:33.990314 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:33.972084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:57039 (local address 127.5.213.254:44623)
0618 22:38:33.972246 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:33.972251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:33.973370 (+ 1119us) server_negotiation.cc:407] Connection header received
0618 22:38:33.975189 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:33.975194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:33.975281 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:33.975367 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:33.977015 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:33.977829 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:33.979387 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:33.979905 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:33.980039 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:33.984263 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:33.984269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:33.984274 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:33.984310 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:33.987394 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:33.988081 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:33.988086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:33.988090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:33.988237 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:33.988670 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:33.988675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:33.988679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:33.988996 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:33.989157 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:33.989763 (+ 606us) server_negotiation.cc:299] Negotiation successful
0618 22:38:33.989986 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:38:33.991108 6873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:33.972541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:44623 (local address 127.5.213.194:57039)
0618 22:38:33.973214 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:33.973251 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:33.974920 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:33.975598 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:33.975610 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:33.976197 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:33.976857 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:33.976878 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:33.977974 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:33.977978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:33.979264 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:33.979276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:33.980215 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:33.980223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:33.980582 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:33.981316 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:33.981347 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:33.984034 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:33.987517 (+ 3483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:33.987524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:33.987537 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:33.987963 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:33.988346 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:33.988350 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:33.988354 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:33.988563 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:33.989100 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:33.989105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:33.989412 (+ 307us) client_negotiation.cc:769] Sending connection context
0618 22:38:33.989626 (+ 214us) client_negotiation.cc:240] Negotiation successful
0618 22:38:33.990000 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":246,"threads_started":1}
WARNING: no policy specified for kudu/127.5.213.195@KRBTEST.COM; defaulting to no policy
I20260618 22:38:33.992483 6871 heartbeater.cc:347] Connected to a master server at 127.5.213.254:44623
I20260618 22:38:33.992741 6871 heartbeater.cc:464] Registering TS with master...
I20260618 22:38:33.993470 6871 heartbeater.cc:511] Master 127.5.213.254:44623 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.195@KRBTEST.COM" created.
I20260618 22:38:33.994838 6548 ts_manager.cc:194] Registered new tserver with Master: ca6dabf36e3249208ea8a3b4f2fae3c2 (127.5.213.194:38685)
I20260618 22:38:33.995890 6548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:57039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
I20260618 22:38:34.014266 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44623
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:34.313086 6876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:34.313357 6876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:34.313482 6876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:34.320627 6876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:34.320734 6876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:34.320883 6876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:38:34.330298 6876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44623
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:34.331499 6876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:34.332778 6876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:34.345083 6881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:34.345139 6882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:34.345111 6884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:34.345661 6876 server_base.cc:1034] running on GCE node
I20260618 22:38:34.346285 6876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:34.347223 6876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:34.348413 6876 hybrid_clock.cc:648] HybridClock initialized: now 1781822314348359 us; error 48 us; skew 500 ppm
I20260618 22:38:34.348719 6876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:34 dist-test-slave-lnj3 krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822314, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:34.351641 6876 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:38:34.352939 6876 webserver.cc:460] Webserver started at http://127.5.213.195:46761/ using document root <none> and password file <none>
I20260618 22:38:34.353560 6876 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:34.353652 6876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:34.353894 6876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:34.356858 6876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "16cf46e4202449478437523415a6def9"
format_stamp: "Formatted at 2026-06-18 22:38:34 on dist-test-slave-lnj3"
server_key: "9908258988c2ba944c073febfd87b4d8"
server_key_iv: "ec70db04eaf198fb4d1fb6a12bcfbb83"
server_key_version: "encryptionkey@0"
I20260618 22:38:34.357548 6876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "16cf46e4202449478437523415a6def9"
format_stamp: "Formatted at 2026-06-18 22:38:34 on dist-test-slave-lnj3"
server_key: "9908258988c2ba944c073febfd87b4d8"
server_key_iv: "ec70db04eaf198fb4d1fb6a12bcfbb83"
server_key_version: "encryptionkey@0"
I20260618 22:38:34.362352 6876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260618 22:38:34.365517 6891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:34.366844 6876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:38:34.366984 6876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "16cf46e4202449478437523415a6def9"
format_stamp: "Formatted at 2026-06-18 22:38:34 on dist-test-slave-lnj3"
server_key: "9908258988c2ba944c073febfd87b4d8"
server_key_iv: "ec70db04eaf198fb4d1fb6a12bcfbb83"
server_key_version: "encryptionkey@0"
I20260618 22:38:34.367152 6876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:34.379801 6876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:34.383697 6876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:34.383945 6876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:34.384811 6876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:38:34.386121 6876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:34.386202 6876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:34.386318 6876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:34.386368 6876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:34.415776 6876 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:37145
I20260618 22:38:34.415753 7003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:37145 every 8 connection(s)
I20260618 22:38:34.417110 6876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:38:34.417742 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 6876
I20260618 22:38:34.417923 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:38:34.418423 5975 external_mini_cluster.cc:1428] Setting key b3220fa3a2e890be662d15c1d7ad9ef2
Jun 18 22:38:34 dist-test-slave-lnj3 krb5kdc[6503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822314, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:34.435719 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:34.419560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:48897 (local address 127.5.213.254:44623)
0618 22:38:34.419753 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:34.419758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:34.421010 (+ 1252us) server_negotiation.cc:407] Connection header received
0618 22:38:34.422878 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:34.422883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:34.422964 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:34.423048 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:34.424474 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.425180 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.426685 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.427091 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.427197 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:34.430364 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:34.430369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:34.430373 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:34.430403 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:34.432973 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:34.433613 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:34.433619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:34.433621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:34.433731 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:34.434098 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:34.434104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:34.434107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:34.434367 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:34.434489 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:34.435180 (+ 691us) server_negotiation.cc:299] Negotiation successful
0618 22:38:34.435402 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260618 22:38:34.436684 7006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:34.420184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:44623 (local address 127.5.213.195:48897)
0618 22:38:34.420872 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:34.420918 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:34.422606 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:34.423286 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:34.423295 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:34.423773 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:34.424308 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:34.424323 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.425306 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.425311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:34.426511 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:34.426521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.427294 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.427301 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:34.427518 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:34.428091 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:34.428133 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:34.430165 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:34.433096 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:34.433105 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:34.433119 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:34.433506 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:34.433826 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:34.433829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:34.433832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:34.434001 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:34.434480 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:34.434488 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:34.434845 (+ 357us) client_negotiation.cc:769] Sending connection context
0618 22:38:34.435075 (+ 230us) client_negotiation.cc:240] Negotiation successful
0618 22:38:34.435573 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":263,"threads_started":1}
I20260618 22:38:34.437876 7004 heartbeater.cc:347] Connected to a master server at 127.5.213.254:44623
I20260618 22:38:34.438133 7004 heartbeater.cc:464] Registering TS with master...
I20260618 22:38:34.438789 7004 heartbeater.cc:511] Master 127.5.213.254:44623 requested a full tablet report, sending...
I20260618 22:38:34.440062 6548 ts_manager.cc:194] Registered new tserver with Master: 16cf46e4202449478437523415a6def9 (127.5.213.195:37145)
I20260618 22:38:34.440831 6548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:48897
I20260618 22:38:34.445574 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:38:34.463013 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:34.451309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52092 (local address 127.5.213.254:44623)
0618 22:38:34.451502 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:34.451508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:34.452044 (+ 536us) server_negotiation.cc:407] Connection header received
0618 22:38:34.452199 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:34.452204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:34.452276 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:34.452363 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:34.454154 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.454936 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.456122 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.456513 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.456605 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:34.458620 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:34.458656 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:34.458661 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:34.458698 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:34.460824 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:34.461376 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:34.461379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:34.461382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:34.461511 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:34.461784 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:34.461787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:34.461789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:34.462091 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:34.462265 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:34.462561 (+ 296us) server_negotiation.cc:299] Negotiation successful
0618 22:38:34.462795 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:38:34.468746 6548 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:38:34.475831 6548 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:38:34.497779 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:34.487478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39098 (local address 127.5.213.193:37953)
0618 22:38:34.488118 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:34.488126 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:34.488801 (+ 675us) server_negotiation.cc:407] Connection header received
0618 22:38:34.489181 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:34.489190 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:34.489458 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:34.489615 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:38:34.490531 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.492057 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.494767 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.496254 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.496383 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:34.496587 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:34.497255 (+ 668us) server_negotiation.cc:299] Negotiation successful
0618 22:38:34.497434 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":1163,"threads_started":1}
I20260618 22:38:34.498955 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:34.487495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:37953 (local address 127.0.0.1:39098)
0618 22:38:34.488656 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:34.488715 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:34.488946 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:34.489927 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:34.489935 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:34.489965 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:38:34.490345 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:34.490357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.492235 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.492240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:34.494582 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:34.494595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.496789 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.496795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:34.496998 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:34.497046 (+ 48us) client_negotiation.cc:769] Sending connection context
0618 22:38:34.498358 (+ 1312us) client_negotiation.cc:240] Negotiation successful
0618 22:38:34.498584 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"thread_start_us":787,"threads_started":1}
I20260618 22:38:34.504444 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1015fd65e149f680b09079d130c243 (DEFAULT_TABLE table=test-table [id=35a55a716dc541f098e7c63e25dc56f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:34.506126 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1015fd65e149f680b09079d130c243. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:34.507529 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:34.489067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:38685 (local address 127.0.0.1:38456)
0618 22:38:34.499981 (+ 10914us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:34.499998 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:34.500091 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:34.500944 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:34.500948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:34.500973 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:38:34.501309 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:34.501321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.503125 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.503130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:34.505035 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:34.505048 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.506779 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.506786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:34.506960 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:34.506994 (+ 34us) client_negotiation.cc:769] Sending connection context
0618 22:38:34.507076 (+ 82us) client_negotiation.cc:240] Negotiation successful
0618 22:38:34.507214 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10816,"thread_start_us":11228,"threads_started":1}
I20260618 22:38:34.508682 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:34.489640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38456 (local address 127.5.213.194:38685)
0618 22:38:34.500349 (+ 10709us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:34.500357 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:34.500391 (+ 34us) server_negotiation.cc:407] Connection header received
0618 22:38:34.500482 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:34.500491 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:34.500755 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:34.500951 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:38:34.501503 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.502971 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.505238 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.506535 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.508019 (+ 1484us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:34.508217 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:34.508282 (+ 65us) server_negotiation.cc:299] Negotiation successful
0618 22:38:34.508424 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10540,"thread_start_us":10446,"threads_started":1}
I20260618 22:38:34.510396 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:34.488090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:37145 (local address 127.0.0.1:54766)
0618 22:38:34.499366 (+ 11276us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:34.499386 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:34.499531 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:34.501394 (+ 1863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:34.501399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:34.501420 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:38:34.501604 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:34.501614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.505384 (+ 3770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.505390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:34.507147 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:34.507162 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.509737 (+ 2575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.509743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:34.509917 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:34.509949 (+ 32us) client_negotiation.cc:769] Sending connection context
0618 22:38:34.510008 (+ 59us) client_negotiation.cc:240] Negotiation successful
0618 22:38:34.510136 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11151,"thread_start_us":11647,"threads_started":1}
I20260618 22:38:34.511471 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:34.488674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54766 (local address 127.5.213.195:37145)
0618 22:38:34.500873 (+ 12199us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:34.500881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:34.500910 (+ 29us) server_negotiation.cc:407] Connection header received
0618 22:38:34.500999 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:34.501005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:34.501217 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:34.501393 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:38:34.502815 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.505239 (+ 2424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.507362 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.508684 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.510726 (+ 2042us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:34.510929 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:34.510997 (+ 68us) server_negotiation.cc:299] Negotiation successful
0618 22:38:34.511141 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12040,"thread_start_us":11891,"threads_started":1}
I20260618 22:38:34.512352 6806 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1015fd65e149f680b09079d130c243 (DEFAULT_TABLE table=test-table [id=35a55a716dc541f098e7c63e25dc56f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:34.513664 6806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1015fd65e149f680b09079d130c243. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:34.514237 6939 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1015fd65e149f680b09079d130c243 (DEFAULT_TABLE table=test-table [id=35a55a716dc541f098e7c63e25dc56f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:34.515281 6939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1015fd65e149f680b09079d130c243. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:34.520570 7022 tablet_bootstrap.cc:492] T 3c1015fd65e149f680b09079d130c243 P e69069ebd26e4cb1a4dd83099d3214bb: Bootstrap starting.
I20260618 22:38:34.524056 7022 tablet_bootstrap.cc:654] T 3c1015fd65e149f680b09079d130c243 P e69069ebd26e4cb1a4dd83099d3214bb: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:34.525254 7022 log.cc:824] T 3c1015fd65e149f680b09079d130c243 P e69069ebd26e4cb1a4dd83099d3214bb: Log is configured to *not* fsync() on all Append() calls
I20260618 22:38:34.526139 7023 tablet_bootstrap.cc:492] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2: Bootstrap starting.
I20260618 22:38:34.527472 7025 tablet_bootstrap.cc:492] T 3c1015fd65e149f680b09079d130c243 P 16cf46e4202449478437523415a6def9: Bootstrap starting.
I20260618 22:38:34.529479 7023 tablet_bootstrap.cc:654] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:34.530215 7022 tablet_bootstrap.cc:492] T 3c1015fd65e149f680b09079d130c243 P e69069ebd26e4cb1a4dd83099d3214bb: No bootstrap required, opened a new log
I20260618 22:38:34.530438 7022 ts_tablet_manager.cc:1397] T 3c1015fd65e149f680b09079d130c243 P e69069ebd26e4cb1a4dd83099d3214bb: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260618 22:38:34.530648 7025 tablet_bootstrap.cc:654] T 3c1015fd65e149f680b09079d130c243 P 16cf46e4202449478437523415a6def9: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:34.530675 7023 log.cc:824] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2: Log is configured to *not* fsync() on all Append() calls
I20260618 22:38:34.531675 7025 log.cc:824] T 3c1015fd65e149f680b09079d130c243 P 16cf46e4202449478437523415a6def9: Log is configured to *not* fsync() on all Append() calls
I20260618 22:38:34.533537 7023 tablet_bootstrap.cc:492] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2: No bootstrap required, opened a new log
I20260618 22:38:34.533790 7023 ts_tablet_manager.cc:1397] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260618 22:38:34.534670 7025 tablet_bootstrap.cc:492] T 3c1015fd65e149f680b09079d130c243 P 16cf46e4202449478437523415a6def9: No bootstrap required, opened a new log
I20260618 22:38:34.534938 7025 ts_tablet_manager.cc:1397] T 3c1015fd65e149f680b09079d130c243 P 16cf46e4202449478437523415a6def9: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260618 22:38:34.535785 6737 heartbeater.cc:503] Master 127.5.213.254:44623 was elected leader, sending a full tablet report...
I20260618 22:38:34.539001 7022 raft_consensus.cc:348] T 3c1015fd65e149f680b09079d130c243 P e69069ebd26e4cb1a4dd83099d3214bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69069ebd26e4cb1a4dd83099d3214bb" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 37953 } } peers { permanent_uuid: "ca6dabf36e3249208ea8a3b4f2fae3c2" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38685 } } peers { permanent_uuid: "16cf46e4202449478437523415a6def9" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37145 } }
I20260618 22:38:34.540213 7022 raft_consensus.cc:374] T 3c1015fd65e149f680b09079d130c243 P e69069ebd26e4cb1a4dd83099d3214bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:34.540323 7022 raft_consensus.cc:729] T 3c1015fd65e149f680b09079d130c243 P e69069ebd26e4cb1a4dd83099d3214bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69069ebd26e4cb1a4dd83099d3214bb, State: Initialized, Role: FOLLOWER
I20260618 22:38:34.540876 7022 consensus_queue.cc:260] T 3c1015fd65e149f680b09079d130c243 P e69069ebd26e4cb1a4dd83099d3214bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69069ebd26e4cb1a4dd83099d3214bb" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 37953 } } peers { permanent_uuid: "ca6dabf36e3249208ea8a3b4f2fae3c2" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38685 } } peers { permanent_uuid: "16cf46e4202449478437523415a6def9" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37145 } }
I20260618 22:38:34.542481 7022 ts_tablet_manager.cc:1428] T 3c1015fd65e149f680b09079d130c243 P e69069ebd26e4cb1a4dd83099d3214bb: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20260618 22:38:34.545120 7023 raft_consensus.cc:348] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69069ebd26e4cb1a4dd83099d3214bb" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 37953 } } peers { permanent_uuid: "ca6dabf36e3249208ea8a3b4f2fae3c2" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38685 } } peers { permanent_uuid: "16cf46e4202449478437523415a6def9" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37145 } }
I20260618 22:38:34.545583 7023 raft_consensus.cc:374] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:34.545678 7023 raft_consensus.cc:729] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6dabf36e3249208ea8a3b4f2fae3c2, State: Initialized, Role: FOLLOWER
I20260618 22:38:34.545481 7025 raft_consensus.cc:348] T 3c1015fd65e149f680b09079d130c243 P 16cf46e4202449478437523415a6def9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69069ebd26e4cb1a4dd83099d3214bb" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 37953 } } peers { permanent_uuid: "ca6dabf36e3249208ea8a3b4f2fae3c2" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38685 } } peers { permanent_uuid: "16cf46e4202449478437523415a6def9" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37145 } }
I20260618 22:38:34.545883 7025 raft_consensus.cc:374] T 3c1015fd65e149f680b09079d130c243 P 16cf46e4202449478437523415a6def9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:34.545982 7025 raft_consensus.cc:729] T 3c1015fd65e149f680b09079d130c243 P 16cf46e4202449478437523415a6def9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16cf46e4202449478437523415a6def9, State: Initialized, Role: FOLLOWER
I20260618 22:38:34.546283 7023 consensus_queue.cc:260] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69069ebd26e4cb1a4dd83099d3214bb" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 37953 } } peers { permanent_uuid: "ca6dabf36e3249208ea8a3b4f2fae3c2" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38685 } } peers { permanent_uuid: "16cf46e4202449478437523415a6def9" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37145 } }
I20260618 22:38:34.546525 7025 consensus_queue.cc:260] T 3c1015fd65e149f680b09079d130c243 P 16cf46e4202449478437523415a6def9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69069ebd26e4cb1a4dd83099d3214bb" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 37953 } } peers { permanent_uuid: "ca6dabf36e3249208ea8a3b4f2fae3c2" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38685 } } peers { permanent_uuid: "16cf46e4202449478437523415a6def9" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37145 } }
I20260618 22:38:34.547408 6871 heartbeater.cc:503] Master 127.5.213.254:44623 was elected leader, sending a full tablet report...
I20260618 22:38:34.548100 7025 ts_tablet_manager.cc:1428] T 3c1015fd65e149f680b09079d130c243 P 16cf46e4202449478437523415a6def9: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260618 22:38:34.548646 7023 ts_tablet_manager.cc:1428] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260618 22:38:34.548650 7004 heartbeater.cc:503] Master 127.5.213.254:44623 was elected leader, sending a full tablet report...
W20260618 22:38:34.668776 7005 tablet.cc:2369] T 3c1015fd65e149f680b09079d130c243 P 16cf46e4202449478437523415a6def9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:38:34.721575 6872 tablet.cc:2369] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:38:34.752928 7029 raft_consensus.cc:482] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:38:34.753207 7029 raft_consensus.cc:504] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69069ebd26e4cb1a4dd83099d3214bb" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 37953 } } peers { permanent_uuid: "ca6dabf36e3249208ea8a3b4f2fae3c2" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38685 } } peers { permanent_uuid: "16cf46e4202449478437523415a6def9" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37145 } }
I20260618 22:38:34.754932 7029 leader_election.cc:288] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e69069ebd26e4cb1a4dd83099d3214bb (127.5.213.193:37953), 16cf46e4202449478437523415a6def9 (127.5.213.195:37145)
W20260618 22:38:34.755937 6738 tablet.cc:2369] T 3c1015fd65e149f680b09079d130c243 P e69069ebd26e4cb1a4dd83099d3214bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:38:34.762048 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:34.755397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:56879 (local address 127.5.213.195:37145)
0618 22:38:34.755560 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:34.755566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:34.757093 (+ 1527us) server_negotiation.cc:407] Connection header received
0618 22:38:34.757271 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:34.757276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:34.757331 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:34.757397 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:38:34.757954 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.758731 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.760498 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.761228 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.761365 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:34.761433 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:34.761713 (+ 280us) server_negotiation.cc:299] Negotiation successful
0618 22:38:34.761822 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 22:38:34.762084 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:34.755325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:47291 (local address 127.5.213.193:37953)
0618 22:38:34.755520 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:34.755526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:34.757059 (+ 1533us) server_negotiation.cc:407] Connection header received
0618 22:38:34.757235 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:34.757240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:34.757296 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:34.757367 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:38:34.757925 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.758760 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.760514 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.761267 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.761394 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:34.761460 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:34.761749 (+ 289us) server_negotiation.cc:299] Negotiation successful
0618 22:38:34.761861 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260618 22:38:34.762244 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:34.755165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:37145 (local address 127.5.213.194:56879)
0618 22:38:34.756994 (+ 1829us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:34.757012 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:34.757132 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:34.757417 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:34.757421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:34.757442 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:38:34.757826 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:34.757835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.758849 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.758853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:34.760371 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:34.760382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.761432 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.761440 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:34.761575 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:34.761596 (+ 21us) client_negotiation.cc:769] Sending connection context
0618 22:38:34.761674 (+ 78us) client_negotiation.cc:240] Negotiation successful
0618 22:38:34.761788 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1733,"mutex_wait_us":104,"thread_start_us":424,"threads_started":1}
I20260618 22:38:34.762292 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:34.755164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:37953 (local address 127.5.213.194:47291)
0618 22:38:34.756955 (+ 1791us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:34.756970 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:34.757107 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:34.757391 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:34.757395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:34.757417 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:38:34.757790 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:34.757799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.758869 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.758875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:34.760380 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:34.760390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.761462 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.761469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:34.761619 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:34.761640 (+ 21us) client_negotiation.cc:769] Sending connection context
0618 22:38:34.761712 (+ 72us) client_negotiation.cc:240] Negotiation successful
0618 22:38:34.761825 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1662,"mutex_wait_us":33,"thread_start_us":1486,"threads_started":1}
I20260618 22:38:34.763187 6959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1015fd65e149f680b09079d130c243" candidate_uuid: "ca6dabf36e3249208ea8a3b4f2fae3c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16cf46e4202449478437523415a6def9" is_pre_election: true
I20260618 22:38:34.763207 6692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1015fd65e149f680b09079d130c243" candidate_uuid: "ca6dabf36e3249208ea8a3b4f2fae3c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e69069ebd26e4cb1a4dd83099d3214bb" is_pre_election: true
I20260618 22:38:34.763621 6959 raft_consensus.cc:2454] T 3c1015fd65e149f680b09079d130c243 P 16cf46e4202449478437523415a6def9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca6dabf36e3249208ea8a3b4f2fae3c2 in term 0.
I20260618 22:38:34.763613 6692 raft_consensus.cc:2454] T 3c1015fd65e149f680b09079d130c243 P e69069ebd26e4cb1a4dd83099d3214bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca6dabf36e3249208ea8a3b4f2fae3c2 in term 0.
I20260618 22:38:34.764171 6760 leader_election.cc:302] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca6dabf36e3249208ea8a3b4f2fae3c2, e69069ebd26e4cb1a4dd83099d3214bb; no voters:
I20260618 22:38:34.764493 7029 raft_consensus.cc:2781] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:38:34.764606 7029 raft_consensus.cc:482] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:38:34.764694 7029 raft_consensus.cc:3037] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:34.766744 7029 raft_consensus.cc:504] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69069ebd26e4cb1a4dd83099d3214bb" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 37953 } } peers { permanent_uuid: "ca6dabf36e3249208ea8a3b4f2fae3c2" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38685 } } peers { permanent_uuid: "16cf46e4202449478437523415a6def9" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37145 } }
I20260618 22:38:34.767299 7029 leader_election.cc:288] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2 [CANDIDATE]: Term 1 election: Requested vote from peers e69069ebd26e4cb1a4dd83099d3214bb (127.5.213.193:37953), 16cf46e4202449478437523415a6def9 (127.5.213.195:37145)
I20260618 22:38:34.767705 6692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1015fd65e149f680b09079d130c243" candidate_uuid: "ca6dabf36e3249208ea8a3b4f2fae3c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e69069ebd26e4cb1a4dd83099d3214bb"
I20260618 22:38:34.767743 6959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1015fd65e149f680b09079d130c243" candidate_uuid: "ca6dabf36e3249208ea8a3b4f2fae3c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16cf46e4202449478437523415a6def9"
I20260618 22:38:34.767901 6692 raft_consensus.cc:3037] T 3c1015fd65e149f680b09079d130c243 P e69069ebd26e4cb1a4dd83099d3214bb [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:34.767905 6959 raft_consensus.cc:3037] T 3c1015fd65e149f680b09079d130c243 P 16cf46e4202449478437523415a6def9 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:34.770908 6692 raft_consensus.cc:2454] T 3c1015fd65e149f680b09079d130c243 P e69069ebd26e4cb1a4dd83099d3214bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca6dabf36e3249208ea8a3b4f2fae3c2 in term 1.
I20260618 22:38:34.770908 6959 raft_consensus.cc:2454] T 3c1015fd65e149f680b09079d130c243 P 16cf46e4202449478437523415a6def9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca6dabf36e3249208ea8a3b4f2fae3c2 in term 1.
I20260618 22:38:34.771246 6761 leader_election.cc:302] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16cf46e4202449478437523415a6def9, ca6dabf36e3249208ea8a3b4f2fae3c2; no voters:
I20260618 22:38:34.771545 7029 raft_consensus.cc:2781] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:38:34.772001 7029 raft_consensus.cc:686] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2 [term 1 LEADER]: Becoming Leader. State: Replica: ca6dabf36e3249208ea8a3b4f2fae3c2, State: Running, Role: LEADER
I20260618 22:38:34.772501 7029 consensus_queue.cc:237] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69069ebd26e4cb1a4dd83099d3214bb" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 37953 } } peers { permanent_uuid: "ca6dabf36e3249208ea8a3b4f2fae3c2" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38685 } } peers { permanent_uuid: "16cf46e4202449478437523415a6def9" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37145 } }
I20260618 22:38:34.776731 6548 catalog_manager.cc:5495] T 3c1015fd65e149f680b09079d130c243 P ca6dabf36e3249208ea8a3b4f2fae3c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca6dabf36e3249208ea8a3b4f2fae3c2 (127.5.213.194). New cstate: current_term: 1 leader_uuid: "ca6dabf36e3249208ea8a3b4f2fae3c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e69069ebd26e4cb1a4dd83099d3214bb" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 37953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca6dabf36e3249208ea8a3b4f2fae3c2" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16cf46e4202449478437523415a6def9" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37145 } health_report { overall_health: UNKNOWN } } }
Jun 18 22:38:34 dist-test-slave-lnj3 krb5kdc[6503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:38:34.865849 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:34.851837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.5.213.193:37953)
0618 22:38:34.852030 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:34.852038 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:34.852430 (+ 392us) server_negotiation.cc:407] Connection header received
0618 22:38:34.852596 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:34.852600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:34.852685 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:34.852772 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:34.854186 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.854960 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.856195 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.856620 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.856745 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:34.860542 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:34.860557 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:34.860574 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:34.860608 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:34.863281 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:34.863946 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:34.863954 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:34.863956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:34.864085 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:34.864430 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:34.864434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:34.864436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:34.864822 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:34.865091 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:34.865382 (+ 291us) server_negotiation.cc:299] Negotiation successful
0618 22:38:34.865552 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Jun 18 22:38:34 dist-test-slave-lnj3 krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822314, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 22:38:34 dist-test-slave-lnj3 krb5kdc[6503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822314, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:38:34.907840 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:34.894923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39128 (local address 127.5.213.193:37953)
0618 22:38:34.895147 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:34.895154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:34.895329 (+ 175us) server_negotiation.cc:407] Connection header received
0618 22:38:34.895513 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:34.895517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:34.895579 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:34.895665 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:34.896963 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.897694 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.899010 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.899416 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.899525 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:34.903096 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:34.903100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:34.903104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:34.903146 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:34.905424 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:34.906010 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:34.906015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:34.906017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:34.906145 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:34.906512 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:34.906516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:34.906518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:34.906833 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:34.907062 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:34.907338 (+ 276us) server_negotiation.cc:299] Negotiation successful
0618 22:38:34.907542 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Jun 18 22:38:34 dist-test-slave-lnj3 krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 22:38:34 dist-test-slave-lnj3 krb5kdc[6503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:38:34.952338 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:34.937280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39134 (local address 127.5.213.193:37953)
0618 22:38:34.937504 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:34.937511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:34.938061 (+ 550us) server_negotiation.cc:407] Connection header received
0618 22:38:34.938237 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:34.938241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:34.938306 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:34.938382 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:34.940004 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.940823 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.942539 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.943005 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.943102 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:34.947479 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:34.947485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:34.947488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:34.947558 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:34.950025 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:34.950703 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:34.950707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:34.950709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:34.950820 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:34.951120 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:34.951123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:34.951124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:34.951406 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:34.951585 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:34.951870 (+ 285us) server_negotiation.cc:299] Negotiation successful
0618 22:38:34.952035 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260618 22:38:34.953976 6652 tablet_service.cc:2908] Rejecting scan request for tablet 3c1015fd65e149f680b09079d130c243: Uninitialized: clean time has not yet been initialized
I20260618 22:38:34.956709 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 6609
I20260618 22:38:34.968289 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:34.957555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39150 (local address 127.5.213.193:37953)
0618 22:38:34.957712 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:34.957717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:34.958024 (+ 307us) server_negotiation.cc:407] Connection header received
0618 22:38:34.958174 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:34.958180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:34.958253 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:34.958345 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:34.959465 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.960113 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.961254 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:34.961648 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:34.961758 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:34.963851 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:34.963857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:34.963860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:34.963887 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:34.965951 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:34.966518 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:34.966523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:34.966527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:34.966667 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:34.966965 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:34.966970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:34.966974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:34.967255 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:34.967511 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:34.967767 (+ 256us) server_negotiation.cc:299] Negotiation successful
0618 22:38:34.967998 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:38:34.968904 6732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:38:35.063913 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 6609
I20260618 22:38:35.076520 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 6743
Jun 18 22:38:35 dist-test-slave-lnj3 krb5kdc[6503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.194@KRBTEST.COM
I20260618 22:38:35.090978 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:35.077302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.5.213.194:38685)
0618 22:38:35.077749 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:35.077753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:35.077770 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:38:35.077820 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:35.077823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:35.077887 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:35.077972 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:35.079305 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:35.080068 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:35.081303 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:35.081672 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:35.081759 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:35.085534 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:35.085548 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:35.085564 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:35.085597 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:35.088495 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:35.089225 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:35.089233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:35.089235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:35.089373 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:35.089741 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:35.089745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:35.089747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:35.090069 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:35.090284 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:35.090503 (+ 219us) server_negotiation.cc:299] Negotiation successful
0618 22:38:35.090720 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":230,"threads_started":1}
I20260618 22:38:35.091565 6866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:38:35.177717 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 6743
I20260618 22:38:35.190356 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 6876
Jun 18 22:38:35 dist-test-slave-lnj3 krb5kdc[6503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.195@KRBTEST.COM
I20260618 22:38:35.203955 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:35.191045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54780 (local address 127.5.213.195:37145)
0618 22:38:35.191208 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:35.191214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:35.191271 (+ 57us) server_negotiation.cc:407] Connection header received
0618 22:38:35.191465 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:35.191471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:35.191531 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:35.191607 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:35.192758 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:35.193452 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:35.194753 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:35.195137 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:35.195243 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:35.198982 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:35.198994 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:35.199009 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:35.199040 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:35.201469 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:35.202048 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:35.202055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:35.202057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:35.202177 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:35.202496 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:35.202500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:35.202502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:35.202986 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:35.203219 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:35.203450 (+ 231us) server_negotiation.cc:299] Negotiation successful
0618 22:38:35.203652 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:38:35.204532 6999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:38:35.288669 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 6876
I20260618 22:38:35.300846 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 6516
I20260618 22:38:35.301993 6578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:38:35.404428 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 6516
2026-06-18T22:38:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4015 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:38:35 dist-test-slave-lnj3 krb5kdc[7053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:38:35 dist-test-slave-lnj3 krb5kdc[7053](info): set up 2 sockets
Jun 18 22:38:35 dist-test-slave-lnj3 krb5kdc[7053](info): commencing operation
krb5kdc: starting...
W20260618 22:38:36.386179 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.007s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:38:36 dist-test-slave-lnj3 krb5kdc[7053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:38:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:38:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:38:36.544072 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:34919
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:34919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:36.843461 7066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:36.843727 7066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:36.843838 7066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:36.850749 7066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:38:36.850835 7066 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260618 22:38:36.850894 7066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:36.850976 7066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:36.851035 7066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:38:36.851109 7066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:38:36.860638 7066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:34919
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:34919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:36.861811 7066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:36.863061 7066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:36.873677 7071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:36.873756 7072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:36.874075 7074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:36.875128 7066 server_base.cc:1034] running on GCE node
I20260618 22:38:36.875887 7066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:36.877179 7066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:36.878409 7066 hybrid_clock.cc:648] HybridClock initialized: now 1781822316878324 us; error 47 us; skew 500 ppm
I20260618 22:38:36.878784 7066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:36 dist-test-slave-lnj3 krb5kdc[7053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822316, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:36.881660 7066 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:38:36.883028 7066 webserver.cc:460] Webserver started at http://127.5.213.254:34069/ using document root <none> and password file <none>
I20260618 22:38:36.883677 7066 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:36.883770 7066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:36.884052 7066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:36.886876 7066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "84e623e3a8a745939d291b8c6e811e6a"
format_stamp: "Formatted at 2026-06-18 22:38:36 on dist-test-slave-lnj3"
server_key: "3ece777c830c251bba4b5c77faf05790"
server_key_iv: "6b9dabf696d649661db07037c034c502"
server_key_version: "encryptionkey@0"
I20260618 22:38:36.887609 7066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "84e623e3a8a745939d291b8c6e811e6a"
format_stamp: "Formatted at 2026-06-18 22:38:36 on dist-test-slave-lnj3"
server_key: "3ece777c830c251bba4b5c77faf05790"
server_key_iv: "6b9dabf696d649661db07037c034c502"
server_key_version: "encryptionkey@0"
I20260618 22:38:36.892422 7066 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260618 22:38:36.895650 7081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:36.896885 7066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:38:36.897037 7066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "84e623e3a8a745939d291b8c6e811e6a"
format_stamp: "Formatted at 2026-06-18 22:38:36 on dist-test-slave-lnj3"
server_key: "3ece777c830c251bba4b5c77faf05790"
server_key_iv: "6b9dabf696d649661db07037c034c502"
server_key_version: "encryptionkey@0"
I20260618 22:38:36.897217 7066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:36.909368 7066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:36.912919 7066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:36.913149 7066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:36.932001 7066 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:34919
I20260618 22:38:36.932020 7142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:34919 every 8 connection(s)
I20260618 22:38:36.933287 7066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:38:36.936811 7143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:36.938740 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 7066
I20260618 22:38:36.938951 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:38:36.939713 5975 external_mini_cluster.cc:1428] Setting key 14e45d56a9260f319061765dd0da7dba
I20260618 22:38:36.949313 7143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a: Bootstrap starting.
I20260618 22:38:36.952566 7143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:36.953460 7143 log.cc:824] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a: Log is configured to *not* fsync() on all Append() calls
Jun 18 22:38:36 dist-test-slave-lnj3 krb5kdc[7053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:36.956143 7143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a: No bootstrap required, opened a new log
I20260618 22:38:36.962690 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:36.941789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36466 (local address 127.5.213.254:34919)
0618 22:38:36.942913 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:36.942928 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:36.943292 (+ 364us) server_negotiation.cc:407] Connection header received
0618 22:38:36.945388 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:36.945419 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:36.945909 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:36.946343 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:36.947544 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:36.949401 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:36.950588 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:36.951603 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:36.951774 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:36.955352 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:36.955362 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:36.955380 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:36.955426 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:36.958982 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:36.959583 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:36.959592 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:36.959598 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:36.959749 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:36.960086 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:36.960090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:36.960092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:36.960606 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:36.960891 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:36.961304 (+ 413us) server_negotiation.cc:299] Negotiation successful
0618 22:38:36.961671 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":633,"threads_started":1}
I20260618 22:38:36.964424 7143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e623e3a8a745939d291b8c6e811e6a" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 34919 } }
I20260618 22:38:36.964859 7143 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:36.964963 7143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84e623e3a8a745939d291b8c6e811e6a, State: Initialized, Role: FOLLOWER
I20260618 22:38:36.965569 7143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e623e3a8a745939d291b8c6e811e6a" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 34919 } }
I20260618 22:38:36.965803 7143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:38:36.965899 7143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:38:36.966010 7143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:36.968147 7143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e623e3a8a745939d291b8c6e811e6a" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 34919 } }
I20260618 22:38:36.968631 7143 leader_election.cc:302] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84e623e3a8a745939d291b8c6e811e6a; no voters:
I20260618 22:38:36.969156 7143 leader_election.cc:288] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:38:36.969274 7148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:38:36.969843 7148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a [term 1 LEADER]: Becoming Leader. State: Replica: 84e623e3a8a745939d291b8c6e811e6a, State: Running, Role: LEADER
I20260618 22:38:36.970324 7148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e623e3a8a745939d291b8c6e811e6a" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 34919 } }
I20260618 22:38:36.970953 7143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:38:36.972898 7150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84e623e3a8a745939d291b8c6e811e6a. Latest consensus state: current_term: 1 leader_uuid: "84e623e3a8a745939d291b8c6e811e6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e623e3a8a745939d291b8c6e811e6a" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 34919 } } }
I20260618 22:38:36.973281 7150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a [sys.catalog]: This master's current role is: LEADER
I20260618 22:38:36.973732 7149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84e623e3a8a745939d291b8c6e811e6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e623e3a8a745939d291b8c6e811e6a" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 34919 } } }
I20260618 22:38:36.973961 7149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a [sys.catalog]: This master's current role is: LEADER
I20260618 22:38:36.975386 7157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:38:36.979841 7157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:38:36.988137 7157 catalog_manager.cc:1347] Generated new cluster ID: c54213fcca0547119150e78af057bedc
I20260618 22:38:36.988240 7157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:38:37.006326 7157 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:38:37.008131 7157 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:38:37.020912 7157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a: Generated new TSK 0
I20260618 22:38:37.021931 7157 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
I20260618 22:38:37.075106 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:34919
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:37.377319 7169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:37.377596 7169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:37.377717 7169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:37.384865 7169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:37.384974 7169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:37.385083 7169 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:38:37.385190 7169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:38:37.394779 7169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.213.254:34919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:37.396127 7169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:37.397434 7169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:37.409991 7174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:37.409916 7177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:37.409936 7175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:37.411018 7169 server_base.cc:1034] running on GCE node
I20260618 22:38:37.411621 7169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:37.412514 7169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:37.413728 7169 hybrid_clock.cc:648] HybridClock initialized: now 1781822317413672 us; error 30 us; skew 500 ppm
I20260618 22:38:37.414006 7169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:37 dist-test-slave-lnj3 krb5kdc[7053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822317, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:37.417104 7169 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:38:37.418416 7169 webserver.cc:460] Webserver started at http://127.5.213.193:38535/ using document root <none> and password file <none>
I20260618 22:38:37.419108 7169 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:37.419189 7169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:37.419445 7169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:37.422238 7169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "d064e4c863914ccdbd32eab2be500d53"
format_stamp: "Formatted at 2026-06-18 22:38:37 on dist-test-slave-lnj3"
server_key: "387948dd32bfa9ee8615d37d2efbf1bd"
server_key_iv: "9858515953d412c415ded623b39131e0"
server_key_version: "encryptionkey@0"
I20260618 22:38:37.422993 7169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "d064e4c863914ccdbd32eab2be500d53"
format_stamp: "Formatted at 2026-06-18 22:38:37 on dist-test-slave-lnj3"
server_key: "387948dd32bfa9ee8615d37d2efbf1bd"
server_key_iv: "9858515953d412c415ded623b39131e0"
server_key_version: "encryptionkey@0"
I20260618 22:38:37.427876 7169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260618 22:38:37.431370 7184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:37.432811 7169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:38:37.432961 7169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "d064e4c863914ccdbd32eab2be500d53"
format_stamp: "Formatted at 2026-06-18 22:38:37 on dist-test-slave-lnj3"
server_key: "387948dd32bfa9ee8615d37d2efbf1bd"
server_key_iv: "9858515953d412c415ded623b39131e0"
server_key_version: "encryptionkey@0"
I20260618 22:38:37.433167 7169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:37.457221 7169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:37.461113 7169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:37.461390 7169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:37.464859 7169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:37.464941 7169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:37.465061 7169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:37.465118 7169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:38:37 dist-test-slave-lnj3 krb5kdc[7053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822317, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:37.499339 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:37.465357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:35353 (local address 127.5.213.254:34919)
0618 22:38:37.482988 (+ 17631us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:37.482993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:37.483011 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:38:37.483061 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:37.483064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:37.483124 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:37.483221 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:37.485825 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:37.486782 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:37.488393 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:37.488787 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:37.488892 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:37.492517 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:37.492522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:37.492526 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:37.492560 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:37.495690 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:37.496720 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:37.496726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:37.496730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:37.496846 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:37.497351 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:37.497356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:37.497360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:37.497652 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:37.497788 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:37.498790 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0618 22:38:37.498995 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17525,"thread_start_us":17431,"threads_started":1}
I20260618 22:38:37.500437 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:37.466108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:34919 (local address 127.5.213.193:35353)
0618 22:38:37.479711 (+ 13603us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:37.479779 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:37.482513 (+ 2734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:37.483538 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:37.483554 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:37.484543 (+ 989us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:37.485615 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:37.485646 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:37.486942 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:37.486947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:37.488253 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:37.488266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:37.489001 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:37.489008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:37.489220 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:37.489863 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:37.489886 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:37.492280 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:37.495852 (+ 3572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:37.495862 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:37.495884 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:37.496598 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:37.496942 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:37.496951 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:37.496955 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:37.497203 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:37.497864 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:37.497873 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:37.498355 (+ 482us) client_negotiation.cc:769] Sending connection context
0618 22:38:37.498719 (+ 364us) client_negotiation.cc:240] Negotiation successful
0618 22:38:37.499241 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13239,"thread_start_us":13024,"threads_started":1}
I20260618 22:38:37.517794 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:37.505563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:58145 (local address 127.5.213.254:34919)
0618 22:38:37.505772 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:37.505778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:37.506520 (+ 742us) server_negotiation.cc:407] Connection header received
0618 22:38:37.506745 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:37.506750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:37.506814 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:37.506890 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:37.507806 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:37.508508 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:37.509739 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:37.510125 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:37.510224 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:37.512265 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:37.512271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:37.512274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:37.512307 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:37.515146 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:37.515876 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:37.515881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:37.515885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:37.516011 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:37.516473 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:37.516478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:37.516481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:37.516776 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:37.516915 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:37.517415 (+ 500us) server_negotiation.cc:299] Negotiation successful
0618 22:38:37.517572 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260618 22:38:37.518249 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:37.505878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:34919 (local address 127.5.213.193:58145)
0618 22:38:37.506421 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:37.506437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:37.506581 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:37.506936 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:37.506940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:37.507408 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:37.507682 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:37.507691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:37.508636 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:37.508642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:37.509616 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:37.509626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:37.510285 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:37.510291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:37.510443 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:37.511088 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:37.511111 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:37.512105 (+ 994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:37.515261 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:37.515266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:37.515270 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:37.515782 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:37.516143 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:37.516149 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:37.516153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:37.516350 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:37.516896 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:37.516901 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:37.517157 (+ 256us) client_negotiation.cc:769] Sending connection context
0618 22:38:37.517341 (+ 184us) client_negotiation.cc:240] Negotiation successful
0618 22:38:37.517712 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":295,"threads_started":1}
I20260618 22:38:37.527390 7169 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:44015
I20260618 22:38:37.527410 7304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:44015 every 8 connection(s)
I20260618 22:38:37.528520 7169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:38:37.530393 7305 heartbeater.cc:347] Connected to a master server at 127.5.213.254:34919
I20260618 22:38:37.530587 7305 heartbeater.cc:464] Registering TS with master...
I20260618 22:38:37.531199 7305 heartbeater.cc:511] Master 127.5.213.254:34919 requested a full tablet report, sending...
I20260618 22:38:37.532192 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 7169
I20260618 22:38:37.532398 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:38:37.533125 5975 external_mini_cluster.cc:1428] Setting key 125362f7189583c4ac3ff95704d1db97
I20260618 22:38:37.533229 7098 ts_manager.cc:194] Registered new tserver with Master: d064e4c863914ccdbd32eab2be500d53 (127.5.213.193:44015)
I20260618 22:38:37.534811 7098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:35353
WARNING: no policy specified for kudu/127.5.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
I20260618 22:38:37.569919 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:34919
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:37.871342 7309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:37.871656 7309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:37.871800 7309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:37.879071 7309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:37.879187 7309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:37.879308 7309 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:38:37.879415 7309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:38:37.889004 7309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.213.254:34919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:37.890247 7309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:37.891551 7309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:37.904014 7315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:37.904181 7317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:37.904543 7309 server_base.cc:1034] running on GCE node
W20260618 22:38:37.904629 7314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:37.905203 7309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:37.906029 7309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:37.907231 7309 hybrid_clock.cc:648] HybridClock initialized: now 1781822317907168 us; error 38 us; skew 500 ppm
I20260618 22:38:37.907493 7309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:37 dist-test-slave-lnj3 krb5kdc[7053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822317, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:37.910264 7309 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:38:37.911521 7309 webserver.cc:460] Webserver started at http://127.5.213.194:40065/ using document root <none> and password file <none>
I20260618 22:38:37.912169 7309 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:37.912261 7309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:37.912544 7309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:37.915264 7309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "4b94d403142441999ed516394491a1c6"
format_stamp: "Formatted at 2026-06-18 22:38:37 on dist-test-slave-lnj3"
server_key: "65c5976ea8ee57395bb3f480c91436f9"
server_key_iv: "4f32c6b539501fc7660870771cfbac14"
server_key_version: "encryptionkey@0"
I20260618 22:38:37.915971 7309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "4b94d403142441999ed516394491a1c6"
format_stamp: "Formatted at 2026-06-18 22:38:37 on dist-test-slave-lnj3"
server_key: "65c5976ea8ee57395bb3f480c91436f9"
server_key_iv: "4f32c6b539501fc7660870771cfbac14"
server_key_version: "encryptionkey@0"
I20260618 22:38:37.921054 7309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:38:37.924453 7324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:37.925762 7309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260618 22:38:37.925901 7309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "4b94d403142441999ed516394491a1c6"
format_stamp: "Formatted at 2026-06-18 22:38:37 on dist-test-slave-lnj3"
server_key: "65c5976ea8ee57395bb3f480c91436f9"
server_key_iv: "4f32c6b539501fc7660870771cfbac14"
server_key_version: "encryptionkey@0"
I20260618 22:38:37.926091 7309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:37.949009 7309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:37.952607 7309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:37.952849 7309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:37.955333 7309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:37.955426 7309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:37.955549 7309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:37.955629 7309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:38:37 dist-test-slave-lnj3 krb5kdc[7053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822317, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:37.975272 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:37.955811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:52369 (local address 127.5.213.254:34919)
0618 22:38:37.955966 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:37.955971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:37.957360 (+ 1389us) server_negotiation.cc:407] Connection header received
0618 22:38:37.959402 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:37.959408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:37.959483 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:37.959568 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:37.961581 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:37.962711 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:37.964943 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:37.965501 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:37.965618 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:37.969529 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:37.969533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:37.969537 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:37.969576 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:37.972464 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:37.973109 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:37.973114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:37.973118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:37.973231 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:37.973609 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:37.973614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:37.973618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:37.973881 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:37.974011 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:37.974745 (+ 734us) server_negotiation.cc:299] Negotiation successful
0618 22:38:37.974943 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:38:37.976238 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:37.956479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:34919 (local address 127.5.213.194:52369)
0618 22:38:37.957213 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:37.957249 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:37.959083 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:37.959815 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:37.959824 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:37.960535 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:37.961356 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:37.961392 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:37.962858 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:37.962863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:37.964814 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:37.964826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:37.965766 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:37.965777 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:37.966072 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:37.966859 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:37.966894 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:37.969332 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:37.972588 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:37.972596 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:37.972612 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:37.973002 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:37.973327 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:37.973332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:37.973336 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:37.973505 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:37.974007 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:37.974015 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:37.974372 (+ 357us) client_negotiation.cc:769] Sending connection context
0618 22:38:37.974646 (+ 274us) client_negotiation.cc:240] Negotiation successful
0618 22:38:37.975124 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":316,"threads_started":1}
I20260618 22:38:37.997242 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:37.981328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:45409 (local address 127.5.213.254:34919)
0618 22:38:37.981501 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:37.981507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:37.982525 (+ 1018us) server_negotiation.cc:407] Connection header received
0618 22:38:37.982707 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:37.982712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:37.982797 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:37.982889 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:37.984067 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:37.985249 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:37.987237 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:37.987940 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:37.988125 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:37.990540 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:37.990547 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:37.990552 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:37.990591 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:37.993925 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:37.994791 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:37.994797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:37.994802 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:37.994949 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:37.995361 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:37.995366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:37.995370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:37.995806 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:37.995934 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:37.996638 (+ 704us) server_negotiation.cc:299] Negotiation successful
0618 22:38:37.996866 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 22:38:37.998795 7381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:37.981331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:34919 (local address 127.5.213.194:45409)
0618 22:38:37.982416 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:37.982436 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:37.982571 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:37.983053 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:37.983058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:37.983582 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:37.983915 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:37.983926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:37.985402 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:37.985408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:37.987081 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:37.987095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:37.988384 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:37.988391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:37.988601 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:37.989262 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:37.989285 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:37.990328 (+ 1043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:37.994082 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:37.994087 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:37.994092 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:37.994662 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:37.995092 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:37.995097 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:37.995102 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:37.995251 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:37.996094 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:37.996099 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:37.996345 (+ 246us) client_negotiation.cc:769] Sending connection context
0618 22:38:37.997809 (+ 1464us) client_negotiation.cc:240] Negotiation successful
0618 22:38:37.998203 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":921,"thread_start_us":705,"threads_started":1}
I20260618 22:38:38.017076 7309 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:40375
I20260618 22:38:38.017112 7443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:40375 every 8 connection(s)
I20260618 22:38:38.018213 7309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:38:38.020226 7444 heartbeater.cc:347] Connected to a master server at 127.5.213.254:34919
I20260618 22:38:38.020418 7444 heartbeater.cc:464] Registering TS with master...
I20260618 22:38:38.021013 7444 heartbeater.cc:511] Master 127.5.213.254:34919 requested a full tablet report, sending...
I20260618 22:38:38.022209 7098 ts_manager.cc:194] Registered new tserver with Master: 4b94d403142441999ed516394491a1c6 (127.5.213.194:40375)
I20260618 22:38:38.023021 7098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:52369
I20260618 22:38:38.026770 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 7309
I20260618 22:38:38.026932 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:38:38.027338 5975 external_mini_cluster.cc:1428] Setting key 4fefbd4482c47d137199deaae33e1cd3
WARNING: no policy specified for kudu/127.5.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
I20260618 22:38:38.059787 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:34919
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:38.367575 7448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:38.367859 7448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:38.367962 7448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:38.375193 7448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:38.375301 7448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:38.375391 7448 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:38:38.375468 7448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:38:38.384938 7448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.213.254:34919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:38.386140 7448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:38.387384 7448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:38.399576 7454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:38.399814 7456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:38.399868 7453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:38.400511 7448 server_base.cc:1034] running on GCE node
I20260618 22:38:38.401173 7448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:38.402104 7448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:38.403316 7448 hybrid_clock.cc:648] HybridClock initialized: now 1781822318403234 us; error 60 us; skew 500 ppm
I20260618 22:38:38.403587 7448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:38 dist-test-slave-lnj3 krb5kdc[7053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822318, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:38.406718 7448 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:38:38.408057 7448 webserver.cc:460] Webserver started at http://127.5.213.195:43533/ using document root <none> and password file <none>
I20260618 22:38:38.408699 7448 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:38.408778 7448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:38.409026 7448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:38.411837 7448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "77380576c915450fb8439a43d8760a22"
format_stamp: "Formatted at 2026-06-18 22:38:38 on dist-test-slave-lnj3"
server_key: "9eaf1b6027d3572fde1f528fd091cabc"
server_key_iv: "6f78baa551cf5a4841eb60c14f91fd33"
server_key_version: "encryptionkey@0"
I20260618 22:38:38.412520 7448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "77380576c915450fb8439a43d8760a22"
format_stamp: "Formatted at 2026-06-18 22:38:38 on dist-test-slave-lnj3"
server_key: "9eaf1b6027d3572fde1f528fd091cabc"
server_key_iv: "6f78baa551cf5a4841eb60c14f91fd33"
server_key_version: "encryptionkey@0"
I20260618 22:38:38.417413 7448 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260618 22:38:38.420708 7463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:38.421947 7448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:38:38.422082 7448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "77380576c915450fb8439a43d8760a22"
format_stamp: "Formatted at 2026-06-18 22:38:38 on dist-test-slave-lnj3"
server_key: "9eaf1b6027d3572fde1f528fd091cabc"
server_key_iv: "6f78baa551cf5a4841eb60c14f91fd33"
server_key_version: "encryptionkey@0"
I20260618 22:38:38.422286 7448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:38.438227 7448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:38.442365 7448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:38.442605 7448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:38.445206 7448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:38.445297 7448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:38.445411 7448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:38.445466 7448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:38:38 dist-test-slave-lnj3 krb5kdc[7053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822318, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:38.468806 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:38.445938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:43243 (local address 127.5.213.254:34919)
0618 22:38:38.446133 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:38.446139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:38.447586 (+ 1447us) server_negotiation.cc:407] Connection header received
0618 22:38:38.450056 (+ 2470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:38.450063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:38.450164 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:38.450279 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:38.452435 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.453457 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.455855 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.456423 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.456553 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:38.461358 (+ 4805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:38.461365 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:38.461371 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:38.461417 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:38.464791 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:38.465872 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:38.465878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:38.465882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:38.466031 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:38.466657 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:38.466662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:38.466665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:38.467018 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:38.467153 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:38.468224 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0618 22:38:38.468452 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:38:38.470252 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:38.446513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:34919 (local address 127.5.213.195:43243)
0618 22:38:38.447399 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:38.447445 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:38.449711 (+ 2266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:38.450565 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:38.450578 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:38.451363 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:38.452243 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:38.452278 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.453595 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.453600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:38.455717 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:38.455732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.456794 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.456807 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:38.457129 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:38.458022 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:38.458044 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:38.461068 (+ 3024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:38.464975 (+ 3907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:38.464985 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:38.465004 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:38.465712 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:38.466166 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:38.466170 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:38.466174 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:38.466480 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:38.467227 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:38.467236 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:38.467766 (+ 530us) client_negotiation.cc:769] Sending connection context
0618 22:38:38.468104 (+ 338us) client_negotiation.cc:240] Negotiation successful
0618 22:38:38.468713 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":368,"threads_started":1}
I20260618 22:38:38.484035 7448 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:45381
I20260618 22:38:38.484048 7582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:45381 every 8 connection(s)
I20260618 22:38:38.485474 7448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:38:38.487676 7583 heartbeater.cc:347] Connected to a master server at 127.5.213.254:34919
I20260618 22:38:38.487874 7583 heartbeater.cc:464] Registering TS with master...
I20260618 22:38:38.488531 7583 heartbeater.cc:511] Master 127.5.213.254:34919 requested a full tablet report, sending...
I20260618 22:38:38.489861 7098 ts_manager.cc:194] Registered new tserver with Master: 77380576c915450fb8439a43d8760a22 (127.5.213.195:45381)
I20260618 22:38:38.490314 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:38.476617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:43697 (local address 127.5.213.254:34919)
0618 22:38:38.476798 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:38.476804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:38.477494 (+ 690us) server_negotiation.cc:407] Connection header received
0618 22:38:38.477682 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:38.477686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:38.477749 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:38.477829 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:38.479111 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.480033 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.482163 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.482540 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.482672 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:38.485043 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:38.485048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:38.485052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:38.485082 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:38.487856 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:38.488450 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:38.488455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:38.488459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:38.488580 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:38.488914 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:38.488919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:38.488923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:38.489212 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:38.489345 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:38.489778 (+ 433us) server_negotiation.cc:299] Negotiation successful
0618 22:38:38.490001 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260618 22:38:38.490331 7565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:38.476635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:34919 (local address 127.5.213.195:43697)
0618 22:38:38.477386 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:38.477408 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:38.477566 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:38.477873 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:38.477878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:38.478475 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:38.478932 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:38.478946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.480163 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.480171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:38.482008 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:38.482025 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.482745 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.482752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:38.482933 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:38.483763 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:38.483792 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:38.484827 (+ 1035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:38.487960 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:38.487966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:38.487971 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:38.488351 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:38.488704 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:38.488709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:38.488713 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:38.488824 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:38.489331 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:38.489335 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:38.489508 (+ 173us) client_negotiation.cc:769] Sending connection context
0618 22:38:38.489629 (+ 121us) client_negotiation.cc:240] Negotiation successful
0618 22:38:38.489884 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":392,"threads_started":1}
I20260618 22:38:38.492000 7098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:43243
I20260618 22:38:38.494195 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 7448
I20260618 22:38:38.494356 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:38:38.494797 5975 external_mini_cluster.cc:1428] Setting key b485314a0df97d05f43578a5fabbe096
I20260618 22:38:38.497547 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 22:38:38 dist-test-slave-lnj3 krb5kdc[7053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822318, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 22:38:38 dist-test-slave-lnj3 krb5kdc[7053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822318, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:38.537930 7305 heartbeater.cc:503] Master 127.5.213.254:34919 was elected leader, sending a full tablet report...
I20260618 22:38:38.541100 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:38.526971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36472 (local address 127.5.213.254:34919)
0618 22:38:38.527539 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:38.527545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:38.527566 (+ 21us) server_negotiation.cc:407] Connection header received
0618 22:38:38.527630 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:38.527635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:38.527722 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:38.527833 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:38.529051 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.530149 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.531992 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.532411 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.532513 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:38.536211 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:38.536216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:38.536220 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:38.536252 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:38.538714 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:38.539319 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:38.539323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:38.539326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:38.539440 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:38.539785 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:38.539788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:38.539790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:38.540052 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:38.540284 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:38.540562 (+ 278us) server_negotiation.cc:299] Negotiation successful
0618 22:38:38.540774 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437}
I20260618 22:38:38.544837 7098 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:38:38.551431 7098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:38:38.583757 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:38.563556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34608 (local address 127.5.213.194:40375)
0618 22:38:38.575349 (+ 11793us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:38.575358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:38.575627 (+ 269us) server_negotiation.cc:407] Connection header received
0618 22:38:38.575732 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:38.575740 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:38.576004 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:38.576978 (+ 974us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:38:38.577216 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.578865 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.581537 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.582695 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.582815 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:38.583026 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:38.583319 (+ 293us) server_negotiation.cc:299] Negotiation successful
0618 22:38:38.583474 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11639,"thread_start_us":12076,"threads_started":1}
I20260618 22:38:38.583804 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:38.563236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:40375 (local address 127.0.0.1:34608)
0618 22:38:38.575385 (+ 12149us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:38.575401 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:38.575508 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:38.576254 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:38.576279 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:38.576313 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:38:38.576760 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:38.576772 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.579023 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.579029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:38.581364 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:38.581375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.582896 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.582902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:38.583078 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:38.583130 (+ 52us) client_negotiation.cc:769] Sending connection context
0618 22:38:38.583276 (+ 146us) client_negotiation.cc:240] Negotiation successful
0618 22:38:38.583457 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12036,"thread_start_us":10802,"threads_started":1}
I20260618 22:38:38.585443 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:38.563261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44308 (local address 127.5.213.193:44015)
0618 22:38:38.576201 (+ 12940us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:38.576210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:38.576235 (+ 25us) server_negotiation.cc:407] Connection header received
0618 22:38:38.576338 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:38.576346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:38.576586 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:38.577478 (+ 892us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:38:38.578864 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.580464 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.582979 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.584420 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.584550 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:38.584744 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:38.585018 (+ 274us) server_negotiation.cc:299] Negotiation successful
0618 22:38:38.585185 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12805,"thread_start_us":12683,"threads_started":1}
I20260618 22:38:38.585966 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:38.563086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:44015 (local address 127.0.0.1:44308)
0618 22:38:38.574537 (+ 11451us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:38.574587 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:38.574822 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:38.576804 (+ 1982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:38.576813 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:38.576840 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:38:38.577277 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:38.577288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.580602 (+ 3314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.580607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:38.582786 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:38.582800 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.584632 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.584638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:38.584853 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:38.584888 (+ 35us) client_negotiation.cc:769] Sending connection context
0618 22:38:38.584973 (+ 85us) client_negotiation.cc:240] Negotiation successful
0618 22:38:38.585091 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11119,"mutex_wait_us":42,"thread_start_us":12041,"threads_started":1}
I20260618 22:38:38.588002 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:38.576280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:45381 (local address 127.0.0.1:35124)
0618 22:38:38.579899 (+ 3619us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:38.579916 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:38.580042 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:38.580583 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:38.580588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:38.580612 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:38:38.581110 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:38.581120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.583057 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.583062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:38.585219 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:38.585232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.587239 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.587245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:38.587410 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:38.587442 (+ 32us) client_negotiation.cc:769] Sending connection context
0618 22:38:38.587543 (+ 101us) client_negotiation.cc:240] Negotiation successful
0618 22:38:38.587659 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3509,"mutex_wait_us":1,"thread_start_us":3413,"threads_started":1}
I20260618 22:38:38.588027 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:38.575827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35124 (local address 127.5.213.195:45381)
0618 22:38:38.579662 (+ 3835us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:38.579671 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:38.580012 (+ 341us) server_negotiation.cc:407] Connection header received
0618 22:38:38.580185 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:38.580193 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:38.580386 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:38.580586 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:38:38.581315 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.582900 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.585377 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.587052 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.587181 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:38.587384 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:38.587584 (+ 200us) server_negotiation.cc:299] Negotiation successful
0618 22:38:38.587735 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3639,"thread_start_us":3476,"threads_started":1}
I20260618 22:38:38.587749 7374 tablet_service.cc:1467] Processing CreateTablet for tablet fbc6419972e0408fa0b3ecc69a436616 (DEFAULT_TABLE table=test-table [id=2785b5bda25941c494fd8b634ec6f69a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:38.589206 7374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbc6419972e0408fa0b3ecc69a436616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:38.589130 7235 tablet_service.cc:1467] Processing CreateTablet for tablet fbc6419972e0408fa0b3ecc69a436616 (DEFAULT_TABLE table=test-table [id=2785b5bda25941c494fd8b634ec6f69a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:38.590461 7235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbc6419972e0408fa0b3ecc69a436616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:38.591634 7513 tablet_service.cc:1467] Processing CreateTablet for tablet fbc6419972e0408fa0b3ecc69a436616 (DEFAULT_TABLE table=test-table [id=2785b5bda25941c494fd8b634ec6f69a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:38.593019 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbc6419972e0408fa0b3ecc69a436616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:38.600296 7602 tablet_bootstrap.cc:492] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6: Bootstrap starting.
I20260618 22:38:38.603467 7602 tablet_bootstrap.cc:654] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:38.605020 7602 log.cc:824] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6: Log is configured to *not* fsync() on all Append() calls
I20260618 22:38:38.605517 7603 tablet_bootstrap.cc:492] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53: Bootstrap starting.
I20260618 22:38:38.606837 7605 tablet_bootstrap.cc:492] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22: Bootstrap starting.
I20260618 22:38:38.609490 7605 tablet_bootstrap.cc:654] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:38.609516 7603 tablet_bootstrap.cc:654] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:38.610414 7605 log.cc:824] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22: Log is configured to *not* fsync() on all Append() calls
I20260618 22:38:38.610747 7603 log.cc:824] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53: Log is configured to *not* fsync() on all Append() calls
I20260618 22:38:38.613173 7605 tablet_bootstrap.cc:492] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22: No bootstrap required, opened a new log
I20260618 22:38:38.613224 7602 tablet_bootstrap.cc:492] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6: No bootstrap required, opened a new log
I20260618 22:38:38.613313 7603 tablet_bootstrap.cc:492] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53: No bootstrap required, opened a new log
I20260618 22:38:38.613432 7605 ts_tablet_manager.cc:1397] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260618 22:38:38.613474 7602 ts_tablet_manager.cc:1397] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260618 22:38:38.613657 7603 ts_tablet_manager.cc:1397] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260618 22:38:38.620210 7605 raft_consensus.cc:348] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:38.620568 7605 raft_consensus.cc:374] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:38.620658 7605 raft_consensus.cc:729] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77380576c915450fb8439a43d8760a22, State: Initialized, Role: FOLLOWER
I20260618 22:38:38.621194 7605 consensus_queue.cc:260] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:38.622202 7583 heartbeater.cc:503] Master 127.5.213.254:34919 was elected leader, sending a full tablet report...
I20260618 22:38:38.622807 7605 ts_tablet_manager.cc:1428] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22: Time spent starting tablet: real 0.009s user 0.007s sys 0.004s
I20260618 22:38:38.623298 7602 raft_consensus.cc:348] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:38.623780 7602 raft_consensus.cc:374] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:38.623878 7602 raft_consensus.cc:729] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b94d403142441999ed516394491a1c6, State: Initialized, Role: FOLLOWER
I20260618 22:38:38.624492 7603 raft_consensus.cc:348] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:38.624577 7602 consensus_queue.cc:260] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:38.624888 7603 raft_consensus.cc:374] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:38.624979 7603 raft_consensus.cc:729] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d064e4c863914ccdbd32eab2be500d53, State: Initialized, Role: FOLLOWER
I20260618 22:38:38.625638 7444 heartbeater.cc:503] Master 127.5.213.254:34919 was elected leader, sending a full tablet report...
I20260618 22:38:38.625599 7603 consensus_queue.cc:260] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:38.626312 7602 ts_tablet_manager.cc:1428] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260618 22:38:38.629745 7603 ts_tablet_manager.cc:1428] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53: Time spent starting tablet: real 0.016s user 0.010s sys 0.003s
W20260618 22:38:38.737036 7584 tablet.cc:2369] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:38:38.750989 7609 raft_consensus.cc:482] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:38:38.751281 7609 raft_consensus.cc:504] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:38.752827 7609 leader_election.cc:288] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d064e4c863914ccdbd32eab2be500d53 (127.5.213.193:44015), 77380576c915450fb8439a43d8760a22 (127.5.213.195:45381)
I20260618 22:38:38.759238 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:38.753263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:58633 (local address 127.5.213.193:44015)
0618 22:38:38.753444 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:38.753449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:38.754006 (+ 557us) server_negotiation.cc:407] Connection header received
0618 22:38:38.754181 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:38.754186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:38.754280 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:38.754385 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:38:38.754923 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.755630 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.757344 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.758389 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.758501 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:38.758553 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:38.758940 (+ 387us) server_negotiation.cc:299] Negotiation successful
0618 22:38:38.759034 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260618 22:38:38.759545 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:38.753205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:44015 (local address 127.5.213.194:58633)
0618 22:38:38.753877 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:38.753896 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:38.754034 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:38.754415 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:38.754419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:38.754452 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:38:38.754791 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:38.754801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.755750 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.755756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:38.757197 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:38.757211 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.758592 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.758599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:38.758785 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:38.758820 (+ 35us) client_negotiation.cc:769] Sending connection context
0618 22:38:38.758918 (+ 98us) client_negotiation.cc:240] Negotiation successful
0618 22:38:38.759051 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":3,"thread_start_us":311,"threads_started":1}
I20260618 22:38:38.760043 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:38.754151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:37101 (local address 127.5.213.195:45381)
0618 22:38:38.754325 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:38.754330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:38.754651 (+ 321us) server_negotiation.cc:407] Connection header received
0618 22:38:38.754806 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:38.754809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:38.754865 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:38.754935 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:38:38.755413 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.756088 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.758103 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.759115 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.759271 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:38.759334 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:38.759746 (+ 412us) server_negotiation.cc:299] Negotiation successful
0618 22:38:38.759864 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260618 22:38:38.760043 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:38.753992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:45381 (local address 127.5.213.194:37101)
0618 22:38:38.754523 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:38.754536 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:38.754683 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:38.754958 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:38.754961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:38.754983 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:38:38.755278 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:38.755286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.756208 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.756214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:38.757943 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:38.757957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.759327 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.759334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:38.759505 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:38.759528 (+ 23us) client_negotiation.cc:769] Sending connection context
0618 22:38:38.759643 (+ 115us) client_negotiation.cc:240] Negotiation successful
0618 22:38:38.759758 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":271,"threads_started":1}
I20260618 22:38:38.760613 7260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc6419972e0408fa0b3ecc69a436616" candidate_uuid: "4b94d403142441999ed516394491a1c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d064e4c863914ccdbd32eab2be500d53" is_pre_election: true
I20260618 22:38:38.760908 7528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc6419972e0408fa0b3ecc69a436616" candidate_uuid: "4b94d403142441999ed516394491a1c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77380576c915450fb8439a43d8760a22" is_pre_election: true
I20260618 22:38:38.761055 7260 raft_consensus.cc:2454] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b94d403142441999ed516394491a1c6 in term 0.
I20260618 22:38:38.761330 7528 raft_consensus.cc:2454] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b94d403142441999ed516394491a1c6 in term 0.
I20260618 22:38:38.761660 7325 leader_election.cc:302] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b94d403142441999ed516394491a1c6, d064e4c863914ccdbd32eab2be500d53; no voters:
I20260618 22:38:38.762051 7609 raft_consensus.cc:2781] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:38:38.762184 7609 raft_consensus.cc:482] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:38:38.762290 7609 raft_consensus.cc:3037] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:38.764628 7609 raft_consensus.cc:504] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:38.765161 7609 leader_election.cc:288] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [CANDIDATE]: Term 1 election: Requested vote from peers d064e4c863914ccdbd32eab2be500d53 (127.5.213.193:44015), 77380576c915450fb8439a43d8760a22 (127.5.213.195:45381)
I20260618 22:38:38.765548 7260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc6419972e0408fa0b3ecc69a436616" candidate_uuid: "4b94d403142441999ed516394491a1c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d064e4c863914ccdbd32eab2be500d53"
I20260618 22:38:38.765648 7528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc6419972e0408fa0b3ecc69a436616" candidate_uuid: "4b94d403142441999ed516394491a1c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77380576c915450fb8439a43d8760a22"
I20260618 22:38:38.765724 7260 raft_consensus.cc:3037] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:38.765841 7528 raft_consensus.cc:3037] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:38.767849 7260 raft_consensus.cc:2454] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b94d403142441999ed516394491a1c6 in term 1.
I20260618 22:38:38.767938 7528 raft_consensus.cc:2454] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b94d403142441999ed516394491a1c6 in term 1.
I20260618 22:38:38.768159 7325 leader_election.cc:302] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b94d403142441999ed516394491a1c6, d064e4c863914ccdbd32eab2be500d53; no voters:
I20260618 22:38:38.768397 7609 raft_consensus.cc:2781] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:38:38.768795 7609 raft_consensus.cc:686] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [term 1 LEADER]: Becoming Leader. State: Replica: 4b94d403142441999ed516394491a1c6, State: Running, Role: LEADER
I20260618 22:38:38.769294 7609 consensus_queue.cc:237] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
W20260618 22:38:38.770011 7445 tablet.cc:2369] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:38:38.773675 7097 catalog_manager.cc:5495] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b94d403142441999ed516394491a1c6 (127.5.213.194). New cstate: current_term: 1 leader_uuid: "4b94d403142441999ed516394491a1c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } health_report { overall_health: UNKNOWN } } }
W20260618 22:38:38.780619 7306 tablet.cc:2369] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:38:38.825462 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:38.820368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34614 (local address 127.5.213.194:40375)
0618 22:38:38.820547 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:38.820553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:38.820570 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:38:38.820865 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:38.820869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:38.820942 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:38.821056 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:38:38.821428 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.822134 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.823438 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:38.823813 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:38.823946 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:38.824413 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:38:38.824885 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:38:38.825012 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:38.825151 (+ 139us) server_negotiation.cc:299] Negotiation successful
0618 22:38:38.825225 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:38:38.832630 7528 raft_consensus.cc:1261] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22 [term 1 FOLLOWER]: Refusing update from remote peer 4b94d403142441999ed516394491a1c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:38:38.832700 7260 raft_consensus.cc:1261] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53 [term 1 FOLLOWER]: Refusing update from remote peer 4b94d403142441999ed516394491a1c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:38:38.833451 7613 consensus_queue.cc:1035] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:38:38.833775 7609 consensus_queue.cc:1035] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:38:38.864080 7098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:38:38.864331 7098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:38:38.867372 7098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a: Sending DeleteTablet for 3 replicas of tablet fbc6419972e0408fa0b3ecc69a436616
I20260618 22:38:38.868430 7374 tablet_service.cc:1514] Processing DeleteTablet for tablet fbc6419972e0408fa0b3ecc69a436616 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:38:38 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:34608
I20260618 22:38:38.868428 7235 tablet_service.cc:1514] Processing DeleteTablet for tablet fbc6419972e0408fa0b3ecc69a436616 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:38:38 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:44308
I20260618 22:38:38.868532 7513 tablet_service.cc:1514] Processing DeleteTablet for tablet fbc6419972e0408fa0b3ecc69a436616 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:38:38 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:35124
I20260618 22:38:38.873159 7098 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:38:38.874647 7098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:38:38.882936 7628 tablet_replica.cc:330] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53: stopping tablet replica
I20260618 22:38:38.883400 7628 raft_consensus.cc:2229] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:38:38.883879 7628 raft_consensus.cc:2258] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:38:38.884641 7626 tablet_replica.cc:330] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6: stopping tablet replica
I20260618 22:38:38.884874 7512 tablet_service.cc:1467] Processing CreateTablet for tablet 752e434f2d7b416da64735237f6d8547 (DEFAULT_TABLE table=test-table [id=84c7f061c5d946018a6aef5535088754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:38.885020 7626 raft_consensus.cc:2229] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:38:38.885094 7234 tablet_service.cc:1467] Processing CreateTablet for tablet 752e434f2d7b416da64735237f6d8547 (DEFAULT_TABLE table=test-table [id=84c7f061c5d946018a6aef5535088754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:38.885427 7512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752e434f2d7b416da64735237f6d8547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:38.885614 7234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752e434f2d7b416da64735237f6d8547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:38.885584 7626 raft_consensus.cc:2258] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:38:38.887558 7626 ts_tablet_manager.cc:1905] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:38:38.891808 7605 tablet_bootstrap.cc:492] T 752e434f2d7b416da64735237f6d8547 P 77380576c915450fb8439a43d8760a22: Bootstrap starting.
I20260618 22:38:38.891745 7374 tablet_service.cc:1467] Processing CreateTablet for tablet 752e434f2d7b416da64735237f6d8547 (DEFAULT_TABLE table=test-table [id=84c7f061c5d946018a6aef5535088754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:38.892357 7374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752e434f2d7b416da64735237f6d8547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:38.893364 7605 tablet_bootstrap.cc:654] T 752e434f2d7b416da64735237f6d8547 P 77380576c915450fb8439a43d8760a22: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:38.893970 7626 ts_tablet_manager.cc:1918] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:38:38.894129 7626 log.cc:1196] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/fbc6419972e0408fa0b3ecc69a436616
I20260618 22:38:38.896613 7628 ts_tablet_manager.cc:1905] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:38:38.899583 7603 tablet_bootstrap.cc:492] T 752e434f2d7b416da64735237f6d8547 P d064e4c863914ccdbd32eab2be500d53: Bootstrap starting.
I20260618 22:38:38.900833 7627 tablet_replica.cc:330] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22: stopping tablet replica
I20260618 22:38:38.901160 7603 tablet_bootstrap.cc:654] T 752e434f2d7b416da64735237f6d8547 P d064e4c863914ccdbd32eab2be500d53: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:38.901291 7627 raft_consensus.cc:2229] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:38:38.901804 7627 raft_consensus.cc:2258] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:38:38.903888 7602 tablet_bootstrap.cc:492] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6: Bootstrap starting.
I20260618 22:38:38.905272 7627 ts_tablet_manager.cc:1905] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:38:38.905416 7602 tablet_bootstrap.cc:654] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:38.908812 7605 tablet_bootstrap.cc:492] T 752e434f2d7b416da64735237f6d8547 P 77380576c915450fb8439a43d8760a22: No bootstrap required, opened a new log
I20260618 22:38:38.908980 7605 ts_tablet_manager.cc:1397] T 752e434f2d7b416da64735237f6d8547 P 77380576c915450fb8439a43d8760a22: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.008s
I20260618 22:38:38.910800 7605 raft_consensus.cc:348] T 752e434f2d7b416da64735237f6d8547 P 77380576c915450fb8439a43d8760a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:38.911079 7605 raft_consensus.cc:374] T 752e434f2d7b416da64735237f6d8547 P 77380576c915450fb8439a43d8760a22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:38.911193 7605 raft_consensus.cc:729] T 752e434f2d7b416da64735237f6d8547 P 77380576c915450fb8439a43d8760a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77380576c915450fb8439a43d8760a22, State: Initialized, Role: FOLLOWER
I20260618 22:38:38.911527 7605 consensus_queue.cc:260] T 752e434f2d7b416da64735237f6d8547 P 77380576c915450fb8439a43d8760a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:38.911854 7603 tablet_bootstrap.cc:492] T 752e434f2d7b416da64735237f6d8547 P d064e4c863914ccdbd32eab2be500d53: No bootstrap required, opened a new log
I20260618 22:38:38.911998 7603 ts_tablet_manager.cc:1397] T 752e434f2d7b416da64735237f6d8547 P d064e4c863914ccdbd32eab2be500d53: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.001s
I20260618 22:38:38.912076 7627 ts_tablet_manager.cc:1918] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:38:38.912249 7627 log.cc:1196] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/fbc6419972e0408fa0b3ecc69a436616
I20260618 22:38:38.912760 7627 ts_tablet_manager.cc:1939] T fbc6419972e0408fa0b3ecc69a436616 P 77380576c915450fb8439a43d8760a22: Deleting consensus metadata
I20260618 22:38:38.912828 7603 raft_consensus.cc:348] T 752e434f2d7b416da64735237f6d8547 P d064e4c863914ccdbd32eab2be500d53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:38.913048 7603 raft_consensus.cc:374] T 752e434f2d7b416da64735237f6d8547 P d064e4c863914ccdbd32eab2be500d53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:38.913147 7603 raft_consensus.cc:729] T 752e434f2d7b416da64735237f6d8547 P d064e4c863914ccdbd32eab2be500d53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d064e4c863914ccdbd32eab2be500d53, State: Initialized, Role: FOLLOWER
I20260618 22:38:38.913380 7603 consensus_queue.cc:260] T 752e434f2d7b416da64735237f6d8547 P d064e4c863914ccdbd32eab2be500d53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:38.913918 7603 ts_tablet_manager.cc:1428] T 752e434f2d7b416da64735237f6d8547 P d064e4c863914ccdbd32eab2be500d53: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260618 22:38:38.914839 7082 catalog_manager.cc:4841] TS 77380576c915450fb8439a43d8760a22 (127.5.213.195:45381): tablet fbc6419972e0408fa0b3ecc69a436616 (table test-table [id=2785b5bda25941c494fd8b634ec6f69a]) successfully deleted
I20260618 22:38:38.915431 7605 ts_tablet_manager.cc:1428] T 752e434f2d7b416da64735237f6d8547 P 77380576c915450fb8439a43d8760a22: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260618 22:38:38.917475 7628 ts_tablet_manager.cc:1918] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:38:38.917616 7628 log.cc:1196] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/fbc6419972e0408fa0b3ecc69a436616
I20260618 22:38:38.918064 7628 ts_tablet_manager.cc:1939] T fbc6419972e0408fa0b3ecc69a436616 P d064e4c863914ccdbd32eab2be500d53: Deleting consensus metadata
I20260618 22:38:38.918752 7626 ts_tablet_manager.cc:1939] T fbc6419972e0408fa0b3ecc69a436616 P 4b94d403142441999ed516394491a1c6: Deleting consensus metadata
I20260618 22:38:38.920774 7085 catalog_manager.cc:4841] TS 4b94d403142441999ed516394491a1c6 (127.5.213.194:40375): tablet fbc6419972e0408fa0b3ecc69a436616 (table test-table [id=2785b5bda25941c494fd8b634ec6f69a]) successfully deleted
I20260618 22:38:38.921859 7082 catalog_manager.cc:4841] TS d064e4c863914ccdbd32eab2be500d53 (127.5.213.193:44015): tablet fbc6419972e0408fa0b3ecc69a436616 (table test-table [id=2785b5bda25941c494fd8b634ec6f69a]) successfully deleted
I20260618 22:38:38.922178 7602 tablet_bootstrap.cc:492] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6: No bootstrap required, opened a new log
I20260618 22:38:38.922338 7602 ts_tablet_manager.cc:1397] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.007s
I20260618 22:38:38.923288 7602 raft_consensus.cc:348] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:38.923765 7602 raft_consensus.cc:374] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:38.924220 7602 raft_consensus.cc:729] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b94d403142441999ed516394491a1c6, State: Initialized, Role: FOLLOWER
I20260618 22:38:38.924538 7602 consensus_queue.cc:260] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:38.925189 7602 ts_tablet_manager.cc:1428] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260618 22:38:39.003636 7613 raft_consensus.cc:482] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:38:39.003897 7613 raft_consensus.cc:504] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:39.004621 7613 leader_election.cc:288] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d064e4c863914ccdbd32eab2be500d53 (127.5.213.193:44015), 77380576c915450fb8439a43d8760a22 (127.5.213.195:45381)
I20260618 22:38:39.005090 7260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752e434f2d7b416da64735237f6d8547" candidate_uuid: "4b94d403142441999ed516394491a1c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d064e4c863914ccdbd32eab2be500d53" is_pre_election: true
I20260618 22:38:39.005193 7528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752e434f2d7b416da64735237f6d8547" candidate_uuid: "4b94d403142441999ed516394491a1c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77380576c915450fb8439a43d8760a22" is_pre_election: true
I20260618 22:38:39.005333 7260 raft_consensus.cc:2454] T 752e434f2d7b416da64735237f6d8547 P d064e4c863914ccdbd32eab2be500d53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b94d403142441999ed516394491a1c6 in term 0.
I20260618 22:38:39.005396 7528 raft_consensus.cc:2454] T 752e434f2d7b416da64735237f6d8547 P 77380576c915450fb8439a43d8760a22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b94d403142441999ed516394491a1c6 in term 0.
I20260618 22:38:39.005787 7325 leader_election.cc:302] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b94d403142441999ed516394491a1c6, d064e4c863914ccdbd32eab2be500d53; no voters:
I20260618 22:38:39.006059 7613 raft_consensus.cc:2781] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:38:39.006162 7613 raft_consensus.cc:482] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:38:39.006264 7613 raft_consensus.cc:3037] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:39.008361 7613 raft_consensus.cc:504] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:39.008919 7613 leader_election.cc:288] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [CANDIDATE]: Term 1 election: Requested vote from peers d064e4c863914ccdbd32eab2be500d53 (127.5.213.193:44015), 77380576c915450fb8439a43d8760a22 (127.5.213.195:45381)
I20260618 22:38:39.009282 7260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752e434f2d7b416da64735237f6d8547" candidate_uuid: "4b94d403142441999ed516394491a1c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d064e4c863914ccdbd32eab2be500d53"
I20260618 22:38:39.009469 7260 raft_consensus.cc:3037] T 752e434f2d7b416da64735237f6d8547 P d064e4c863914ccdbd32eab2be500d53 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:39.009466 7528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752e434f2d7b416da64735237f6d8547" candidate_uuid: "4b94d403142441999ed516394491a1c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77380576c915450fb8439a43d8760a22"
I20260618 22:38:39.009634 7528 raft_consensus.cc:3037] T 752e434f2d7b416da64735237f6d8547 P 77380576c915450fb8439a43d8760a22 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:39.012223 7260 raft_consensus.cc:2454] T 752e434f2d7b416da64735237f6d8547 P d064e4c863914ccdbd32eab2be500d53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b94d403142441999ed516394491a1c6 in term 1.
I20260618 22:38:39.012224 7528 raft_consensus.cc:2454] T 752e434f2d7b416da64735237f6d8547 P 77380576c915450fb8439a43d8760a22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b94d403142441999ed516394491a1c6 in term 1.
I20260618 22:38:39.012573 7325 leader_election.cc:302] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b94d403142441999ed516394491a1c6, d064e4c863914ccdbd32eab2be500d53; no voters:
I20260618 22:38:39.012813 7613 raft_consensus.cc:2781] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:38:39.012964 7613 raft_consensus.cc:686] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [term 1 LEADER]: Becoming Leader. State: Replica: 4b94d403142441999ed516394491a1c6, State: Running, Role: LEADER
I20260618 22:38:39.013196 7613 consensus_queue.cc:237] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:39.015564 7098 catalog_manager.cc:5495] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b94d403142441999ed516394491a1c6 (127.5.213.194). New cstate: current_term: 1 leader_uuid: "4b94d403142441999ed516394491a1c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } health_report { overall_health: UNKNOWN } } }
Jun 18 22:38:39 dist-test-slave-lnj3 krb5kdc[7053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822316, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:39.078920 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:39.064962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:34919 (local address 127.0.0.1:36488)
0618 22:38:39.065736 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:39.065755 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:39.065885 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:39.066217 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:39.066220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:39.066918 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:39.067159 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:39.067176 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.068636 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.068639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:39.069529 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:39.069539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.070423 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.070427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:39.070555 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:39.071187 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:39.071208 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:39.073593 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:39.076317 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:39.076329 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:39.076336 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:39.076854 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:39.077150 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:39.077155 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:39.077159 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:39.077308 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:39.077830 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:39.077839 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:39.078012 (+ 173us) client_negotiation.cc:769] Sending connection context
0618 22:38:39.078176 (+ 164us) client_negotiation.cc:240] Negotiation successful
0618 22:38:39.078440 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":484,"threads_started":1}
I20260618 22:38:39.078949 7639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:39.064998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36488 (local address 127.5.213.254:34919)
0618 22:38:39.065538 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:39.065544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:39.065837 (+ 293us) server_negotiation.cc:407] Connection header received
0618 22:38:39.065981 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:39.065986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:39.066065 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:39.066191 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:39.067314 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.068522 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.069647 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.070271 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.070386 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:39.073790 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:39.073795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:39.073798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:39.073824 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:39.076200 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:39.076947 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:39.076951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:39.076953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:39.077062 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:39.077401 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:39.077404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:39.077406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:39.077723 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:39.077819 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:39.078258 (+ 439us) server_negotiation.cc:299] Negotiation successful
0618 22:38:39.078556 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":371,"thread_start_us":238,"threads_started":1}
I20260618 22:38:39.083951 7098 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:36488:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260618 22:38:39.084928 7098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:38:39.090972 7234 tablet_service.cc:1467] Processing CreateTablet for tablet 817194a6ffe64386ac1c434587a962b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc757a50647a4de3911450076e3ee142]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:38:39.091116 7374 tablet_service.cc:1467] Processing CreateTablet for tablet 817194a6ffe64386ac1c434587a962b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc757a50647a4de3911450076e3ee142]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:38:39.091495 7234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 817194a6ffe64386ac1c434587a962b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:39.091425 7513 tablet_service.cc:1467] Processing CreateTablet for tablet 817194a6ffe64386ac1c434587a962b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc757a50647a4de3911450076e3ee142]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:38:39.091671 7374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 817194a6ffe64386ac1c434587a962b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:39.091900 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 817194a6ffe64386ac1c434587a962b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:39.098328 7605 tablet_bootstrap.cc:492] T 817194a6ffe64386ac1c434587a962b7 P 77380576c915450fb8439a43d8760a22: Bootstrap starting.
I20260618 22:38:39.098409 7602 tablet_bootstrap.cc:492] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6: Bootstrap starting.
I20260618 22:38:39.099623 7605 tablet_bootstrap.cc:654] T 817194a6ffe64386ac1c434587a962b7 P 77380576c915450fb8439a43d8760a22: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:39.099895 7602 tablet_bootstrap.cc:654] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:39.101326 7603 tablet_bootstrap.cc:492] T 817194a6ffe64386ac1c434587a962b7 P d064e4c863914ccdbd32eab2be500d53: Bootstrap starting.
I20260618 22:38:39.102058 7605 tablet_bootstrap.cc:492] T 817194a6ffe64386ac1c434587a962b7 P 77380576c915450fb8439a43d8760a22: No bootstrap required, opened a new log
I20260618 22:38:39.102165 7605 ts_tablet_manager.cc:1397] T 817194a6ffe64386ac1c434587a962b7 P 77380576c915450fb8439a43d8760a22: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260618 22:38:39.102901 7603 tablet_bootstrap.cc:654] T 817194a6ffe64386ac1c434587a962b7 P d064e4c863914ccdbd32eab2be500d53: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:39.102772 7605 raft_consensus.cc:348] T 817194a6ffe64386ac1c434587a962b7 P 77380576c915450fb8439a43d8760a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:39.102998 7605 raft_consensus.cc:374] T 817194a6ffe64386ac1c434587a962b7 P 77380576c915450fb8439a43d8760a22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:39.103067 7605 raft_consensus.cc:729] T 817194a6ffe64386ac1c434587a962b7 P 77380576c915450fb8439a43d8760a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77380576c915450fb8439a43d8760a22, State: Initialized, Role: FOLLOWER
I20260618 22:38:39.103320 7605 consensus_queue.cc:260] T 817194a6ffe64386ac1c434587a962b7 P 77380576c915450fb8439a43d8760a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:39.103485 7602 tablet_bootstrap.cc:492] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6: No bootstrap required, opened a new log
I20260618 22:38:39.103647 7602 ts_tablet_manager.cc:1397] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260618 22:38:39.103746 7608 tablet_replica.cc:439] T 817194a6ffe64386ac1c434587a962b7 P 77380576c915450fb8439a43d8760a22: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } } }
I20260618 22:38:39.104007 7608 tablet_replica.cc:442] T 817194a6ffe64386ac1c434587a962b7 P 77380576c915450fb8439a43d8760a22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:38:39.104360 7602 raft_consensus.cc:348] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:39.104568 7602 raft_consensus.cc:374] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:39.104645 7602 raft_consensus.cc:729] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b94d403142441999ed516394491a1c6, State: Initialized, Role: FOLLOWER
I20260618 22:38:39.104885 7602 consensus_queue.cc:260] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:39.105444 7602 ts_tablet_manager.cc:1428] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260618 22:38:39.105301 7613 tablet_replica.cc:439] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } } }
I20260618 22:38:39.105554 7613 tablet_replica.cc:442] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:38:39.105840 7605 ts_tablet_manager.cc:1428] T 817194a6ffe64386ac1c434587a962b7 P 77380576c915450fb8439a43d8760a22: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260618 22:38:39.111943 7603 tablet_bootstrap.cc:492] T 817194a6ffe64386ac1c434587a962b7 P d064e4c863914ccdbd32eab2be500d53: No bootstrap required, opened a new log
I20260618 22:38:39.112123 7603 ts_tablet_manager.cc:1397] T 817194a6ffe64386ac1c434587a962b7 P d064e4c863914ccdbd32eab2be500d53: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20260618 22:38:39.112848 7603 raft_consensus.cc:348] T 817194a6ffe64386ac1c434587a962b7 P d064e4c863914ccdbd32eab2be500d53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:39.113090 7603 raft_consensus.cc:374] T 817194a6ffe64386ac1c434587a962b7 P d064e4c863914ccdbd32eab2be500d53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:39.113179 7603 raft_consensus.cc:729] T 817194a6ffe64386ac1c434587a962b7 P d064e4c863914ccdbd32eab2be500d53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d064e4c863914ccdbd32eab2be500d53, State: Initialized, Role: FOLLOWER
I20260618 22:38:39.113402 7603 consensus_queue.cc:260] T 817194a6ffe64386ac1c434587a962b7 P d064e4c863914ccdbd32eab2be500d53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:39.114004 7603 ts_tablet_manager.cc:1428] T 817194a6ffe64386ac1c434587a962b7 P d064e4c863914ccdbd32eab2be500d53: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260618 22:38:39.113859 7610 tablet_replica.cc:439] T 817194a6ffe64386ac1c434587a962b7 P d064e4c863914ccdbd32eab2be500d53: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } } }
I20260618 22:38:39.114228 7610 tablet_replica.cc:442] T 817194a6ffe64386ac1c434587a962b7 P d064e4c863914ccdbd32eab2be500d53: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:38:39.285802 7613 raft_consensus.cc:482] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:38:39.286022 7613 raft_consensus.cc:504] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:39.286825 7613 leader_election.cc:288] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d064e4c863914ccdbd32eab2be500d53 (127.5.213.193:44015), 77380576c915450fb8439a43d8760a22 (127.5.213.195:45381)
I20260618 22:38:39.287384 7260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "817194a6ffe64386ac1c434587a962b7" candidate_uuid: "4b94d403142441999ed516394491a1c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d064e4c863914ccdbd32eab2be500d53" is_pre_election: true
I20260618 22:38:39.287412 7528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "817194a6ffe64386ac1c434587a962b7" candidate_uuid: "4b94d403142441999ed516394491a1c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77380576c915450fb8439a43d8760a22" is_pre_election: true
I20260618 22:38:39.287631 7260 raft_consensus.cc:2454] T 817194a6ffe64386ac1c434587a962b7 P d064e4c863914ccdbd32eab2be500d53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b94d403142441999ed516394491a1c6 in term 0.
I20260618 22:38:39.287658 7528 raft_consensus.cc:2454] T 817194a6ffe64386ac1c434587a962b7 P 77380576c915450fb8439a43d8760a22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b94d403142441999ed516394491a1c6 in term 0.
I20260618 22:38:39.288053 7325 leader_election.cc:302] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b94d403142441999ed516394491a1c6, d064e4c863914ccdbd32eab2be500d53; no voters:
I20260618 22:38:39.288301 7613 raft_consensus.cc:2781] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:38:39.288420 7613 raft_consensus.cc:482] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:38:39.288509 7613 raft_consensus.cc:3037] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:39.290555 7613 raft_consensus.cc:504] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:39.291112 7613 leader_election.cc:288] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6 [CANDIDATE]: Term 1 election: Requested vote from peers d064e4c863914ccdbd32eab2be500d53 (127.5.213.193:44015), 77380576c915450fb8439a43d8760a22 (127.5.213.195:45381)
I20260618 22:38:39.291539 7260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "817194a6ffe64386ac1c434587a962b7" candidate_uuid: "4b94d403142441999ed516394491a1c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d064e4c863914ccdbd32eab2be500d53"
I20260618 22:38:39.291622 7528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "817194a6ffe64386ac1c434587a962b7" candidate_uuid: "4b94d403142441999ed516394491a1c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77380576c915450fb8439a43d8760a22"
I20260618 22:38:39.291760 7260 raft_consensus.cc:3037] T 817194a6ffe64386ac1c434587a962b7 P d064e4c863914ccdbd32eab2be500d53 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:39.291842 7528 raft_consensus.cc:3037] T 817194a6ffe64386ac1c434587a962b7 P 77380576c915450fb8439a43d8760a22 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:39.293690 7260 raft_consensus.cc:2454] T 817194a6ffe64386ac1c434587a962b7 P d064e4c863914ccdbd32eab2be500d53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b94d403142441999ed516394491a1c6 in term 1.
I20260618 22:38:39.294073 7325 leader_election.cc:302] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b94d403142441999ed516394491a1c6, d064e4c863914ccdbd32eab2be500d53; no voters:
I20260618 22:38:39.294330 7613 raft_consensus.cc:2781] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:38:39.294477 7613 raft_consensus.cc:686] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6 [term 1 LEADER]: Becoming Leader. State: Replica: 4b94d403142441999ed516394491a1c6, State: Running, Role: LEADER
I20260618 22:38:39.294771 7613 consensus_queue.cc:237] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } }
I20260618 22:38:39.294895 7528 raft_consensus.cc:2454] T 817194a6ffe64386ac1c434587a962b7 P 77380576c915450fb8439a43d8760a22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b94d403142441999ed516394491a1c6 in term 1.
I20260618 22:38:39.296525 7609 tablet_replica.cc:439] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6: TxnStatusTablet state changed. Reason: New leader 4b94d403142441999ed516394491a1c6. Latest consensus state: current_term: 1 leader_uuid: "4b94d403142441999ed516394491a1c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } } }
I20260618 22:38:39.296759 7609 tablet_replica.cc:442] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:38:39.298441 7097 catalog_manager.cc:5495] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b94d403142441999ed516394491a1c6 (127.5.213.194). New cstate: current_term: 1 leader_uuid: "4b94d403142441999ed516394491a1c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } health_report { overall_health: UNKNOWN } } }
I20260618 22:38:39.299197 7644 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260618 22:38:39.299399 7644 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260618 22:38:39.354600 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:39.349049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34616 (local address 127.5.213.194:40375)
0618 22:38:39.349651 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:39.349656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:39.349676 (+ 20us) server_negotiation.cc:407] Connection header received
0618 22:38:39.349732 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:39.349735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:39.349794 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:39.349869 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:38:39.350258 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.351060 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.352878 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.353333 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.353420 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:39.353897 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:38:39.353991 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:38:39.354049 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:39.354202 (+ 153us) server_negotiation.cc:299] Negotiation successful
0618 22:38:39.354286 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":443,"threads_started":1}
I20260618 22:38:39.354610 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:39.348904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:40375 (local address 127.0.0.1:34616)
0618 22:38:39.349164 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:39.349184 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:39.349314 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:39.349918 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:39.349923 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:39.349936 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:38:39.350132 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:39.350142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.351171 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.351177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:39.352737 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:39.352752 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.353473 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.353479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:39.353672 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:39.353744 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:38:39.354067 (+ 323us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:38:39.354083 (+ 16us) client_negotiation.cc:769] Sending connection context
0618 22:38:39.354175 (+ 92us) client_negotiation.cc:240] Negotiation successful
0618 22:38:39.354321 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260618 22:38:39.358220 7260 raft_consensus.cc:1261] T 817194a6ffe64386ac1c434587a962b7 P d064e4c863914ccdbd32eab2be500d53 [term 1 FOLLOWER]: Refusing update from remote peer 4b94d403142441999ed516394491a1c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:38:39.358237 7528 raft_consensus.cc:1261] T 817194a6ffe64386ac1c434587a962b7 P 77380576c915450fb8439a43d8760a22 [term 1 FOLLOWER]: Refusing update from remote peer 4b94d403142441999ed516394491a1c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:38:39.358811 7609 consensus_queue.cc:1035] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:38:39.359068 7613 consensus_queue.cc:1035] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:38:39.367465 7608 tablet_replica.cc:439] T 817194a6ffe64386ac1c434587a962b7 P 77380576c915450fb8439a43d8760a22: TxnStatusTablet state changed. Reason: New leader 4b94d403142441999ed516394491a1c6. Latest consensus state: current_term: 1 leader_uuid: "4b94d403142441999ed516394491a1c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } } }
I20260618 22:38:39.367733 7608 tablet_replica.cc:442] T 817194a6ffe64386ac1c434587a962b7 P 77380576c915450fb8439a43d8760a22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:38:39.369629 7647 mvcc.cc:205] Tried to move back new op lower bound from 7298344220085694464 to 7298344219833675776. Current Snapshot: MvccSnapshot[applied={T|T < 7298344220085694464}]
I20260618 22:38:39.370760 7648 mvcc.cc:205] Tried to move back new op lower bound from 7298344220085694464 to 7298344219833675776. Current Snapshot: MvccSnapshot[applied={T|T < 7298344220085694464}]
I20260618 22:38:39.375104 7608 tablet_replica.cc:439] T 817194a6ffe64386ac1c434587a962b7 P 77380576c915450fb8439a43d8760a22: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b94d403142441999ed516394491a1c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } } }
I20260618 22:38:39.375357 7608 tablet_replica.cc:442] T 817194a6ffe64386ac1c434587a962b7 P 77380576c915450fb8439a43d8760a22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:38:39.383929 7610 tablet_replica.cc:439] T 817194a6ffe64386ac1c434587a962b7 P d064e4c863914ccdbd32eab2be500d53: TxnStatusTablet state changed. Reason: New leader 4b94d403142441999ed516394491a1c6. Latest consensus state: current_term: 1 leader_uuid: "4b94d403142441999ed516394491a1c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } } }
I20260618 22:38:39.384199 7610 tablet_replica.cc:442] T 817194a6ffe64386ac1c434587a962b7 P d064e4c863914ccdbd32eab2be500d53: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:38:39.384467 7613 tablet_replica.cc:439] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b94d403142441999ed516394491a1c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } } }
I20260618 22:38:39.384781 7613 tablet_replica.cc:442] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:38:39.388044 7613 tablet_replica.cc:439] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b94d403142441999ed516394491a1c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } } }
I20260618 22:38:39.388329 7613 tablet_replica.cc:442] T 817194a6ffe64386ac1c434587a962b7 P 4b94d403142441999ed516394491a1c6: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:38:39.389398 7610 tablet_replica.cc:439] T 817194a6ffe64386ac1c434587a962b7 P d064e4c863914ccdbd32eab2be500d53: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b94d403142441999ed516394491a1c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 } } peers { permanent_uuid: "4b94d403142441999ed516394491a1c6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40375 } } peers { permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 } } }
I20260618 22:38:39.389647 7610 tablet_replica.cc:442] T 817194a6ffe64386ac1c434587a962b7 P d064e4c863914ccdbd32eab2be500d53: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:38:39.404361 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:39.399165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:56553 (local address 127.5.213.194:40375)
0618 22:38:39.399324 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:39.399334 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:39.399782 (+ 448us) server_negotiation.cc:407] Connection header received
0618 22:38:39.399930 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:39.399935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:39.400012 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:39.400120 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:38:39.400593 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.401372 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.402818 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.403227 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.403358 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:39.403718 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:38:39.403826 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:38:39.403893 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:39.404003 (+ 110us) server_negotiation.cc:299] Negotiation successful
0618 22:38:39.404065 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:38:39.404474 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:39.399091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:40375 (local address 127.5.213.194:56553)
0618 22:38:39.399695 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:39.399711 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:39.399823 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:39.400141 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:39.400144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:39.400154 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:38:39.400433 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:39.400442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.401469 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.401474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:39.402693 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:39.402705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.403381 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.403387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:39.403527 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:39.403578 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:38:39.403906 (+ 328us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:38:39.403915 (+ 9us) client_negotiation.cc:769] Sending connection context
0618 22:38:39.403983 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 22:38:39.404108 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":277,"threads_started":1}
I20260618 22:38:39.413400 7528 raft_consensus.cc:1261] T 752e434f2d7b416da64735237f6d8547 P 77380576c915450fb8439a43d8760a22 [term 1 FOLLOWER]: Refusing update from remote peer 4b94d403142441999ed516394491a1c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:38:39.413458 7260 raft_consensus.cc:1261] T 752e434f2d7b416da64735237f6d8547 P d064e4c863914ccdbd32eab2be500d53 [term 1 FOLLOWER]: Refusing update from remote peer 4b94d403142441999ed516394491a1c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:38:39.413975 7609 consensus_queue.cc:1035] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77380576c915450fb8439a43d8760a22" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:38:39.414266 7613 consensus_queue.cc:1035] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d064e4c863914ccdbd32eab2be500d53" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:38:39.420022 7649 mvcc.cc:205] Tried to move back new op lower bound from 7298344220312846336 to 7298344218680037376. Current Snapshot: MvccSnapshot[applied={T|T < 7298344220312846336}]
I20260618 22:38:39.469596 7097 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:38:39.469928 7097 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:38:39.475395 7097 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84e623e3a8a745939d291b8c6e811e6a: Sending DeleteTablet for 3 replicas of tablet 752e434f2d7b416da64735237f6d8547
I20260618 22:38:39.476245 7234 tablet_service.cc:1514] Processing DeleteTablet for tablet 752e434f2d7b416da64735237f6d8547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:38:39 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:44308
I20260618 22:38:39.476961 7664 tablet_replica.cc:330] T 752e434f2d7b416da64735237f6d8547 P d064e4c863914ccdbd32eab2be500d53: stopping tablet replica
I20260618 22:38:39.476969 7513 tablet_service.cc:1514] Processing DeleteTablet for tablet 752e434f2d7b416da64735237f6d8547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:38:39 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:35124
I20260618 22:38:39.477912 7373 tablet_service.cc:1514] Processing DeleteTablet for tablet 752e434f2d7b416da64735237f6d8547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:38:39 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:34608
I20260618 22:38:39.477344 7664 raft_consensus.cc:2229] T 752e434f2d7b416da64735237f6d8547 P d064e4c863914ccdbd32eab2be500d53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:38:39.483886 7664 raft_consensus.cc:2258] T 752e434f2d7b416da64735237f6d8547 P d064e4c863914ccdbd32eab2be500d53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:38:39.489135 7665 tablet_replica.cc:330] T 752e434f2d7b416da64735237f6d8547 P 77380576c915450fb8439a43d8760a22: stopping tablet replica
I20260618 22:38:39.489461 7665 raft_consensus.cc:2229] T 752e434f2d7b416da64735237f6d8547 P 77380576c915450fb8439a43d8760a22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:38:39.489755 7665 raft_consensus.cc:2258] T 752e434f2d7b416da64735237f6d8547 P 77380576c915450fb8439a43d8760a22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:38:39.490304 7666 tablet_replica.cc:330] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6: stopping tablet replica
I20260618 22:38:39.490357 7664 ts_tablet_manager.cc:1905] T 752e434f2d7b416da64735237f6d8547 P d064e4c863914ccdbd32eab2be500d53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:38:39.491024 7665 ts_tablet_manager.cc:1905] T 752e434f2d7b416da64735237f6d8547 P 77380576c915450fb8439a43d8760a22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:38:39.494820 7666 raft_consensus.cc:2229] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:38:39.495345 7666 raft_consensus.cc:2258] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:38:39.496419 7666 ts_tablet_manager.cc:1905] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:38:39.497969 7664 ts_tablet_manager.cc:1918] T 752e434f2d7b416da64735237f6d8547 P d064e4c863914ccdbd32eab2be500d53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:38:39.498620 7664 log.cc:1196] T 752e434f2d7b416da64735237f6d8547 P d064e4c863914ccdbd32eab2be500d53: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/752e434f2d7b416da64735237f6d8547
I20260618 22:38:39.499200 7664 ts_tablet_manager.cc:1939] T 752e434f2d7b416da64735237f6d8547 P d064e4c863914ccdbd32eab2be500d53: Deleting consensus metadata
I20260618 22:38:39.500397 7082 catalog_manager.cc:4841] TS d064e4c863914ccdbd32eab2be500d53 (127.5.213.193:44015): tablet 752e434f2d7b416da64735237f6d8547 (table test-table [id=84c7f061c5d946018a6aef5535088754]) successfully deleted
I20260618 22:38:39.506686 7666 ts_tablet_manager.cc:1918] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:38:39.506938 7666 log.cc:1196] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/752e434f2d7b416da64735237f6d8547
I20260618 22:38:39.507472 7666 ts_tablet_manager.cc:1939] T 752e434f2d7b416da64735237f6d8547 P 4b94d403142441999ed516394491a1c6: Deleting consensus metadata
I20260618 22:38:39.507695 7665 ts_tablet_manager.cc:1918] T 752e434f2d7b416da64735237f6d8547 P 77380576c915450fb8439a43d8760a22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:38:39.507889 7665 log.cc:1196] T 752e434f2d7b416da64735237f6d8547 P 77380576c915450fb8439a43d8760a22: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/752e434f2d7b416da64735237f6d8547
I20260618 22:38:39.508394 7665 ts_tablet_manager.cc:1939] T 752e434f2d7b416da64735237f6d8547 P 77380576c915450fb8439a43d8760a22: Deleting consensus metadata
I20260618 22:38:39.508795 7085 catalog_manager.cc:4841] TS 4b94d403142441999ed516394491a1c6 (127.5.213.194:40375): tablet 752e434f2d7b416da64735237f6d8547 (table test-table [id=84c7f061c5d946018a6aef5535088754]) successfully deleted
I20260618 22:38:39.509579 7082 catalog_manager.cc:4841] TS 77380576c915450fb8439a43d8760a22 (127.5.213.195:45381): tablet 752e434f2d7b416da64735237f6d8547 (table test-table [id=84c7f061c5d946018a6aef5535088754]) successfully deleted
Jun 18 22:38:39 dist-test-slave-lnj3 krb5kdc[7053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822318, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:38:39.526964 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:39.480597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44318 (local address 127.5.213.193:44015)
0618 22:38:39.498522 (+ 17925us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:39.498530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:39.507546 (+ 9016us) server_negotiation.cc:407] Connection header received
0618 22:38:39.508133 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:39.508140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:39.508244 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:39.508337 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:39.510544 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.511728 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.513647 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.514293 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.514412 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:39.519566 (+ 5154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:39.519588 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:39.519609 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:39.519659 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:39.523594 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:39.524597 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:39.524609 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:39.524613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:39.524790 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:39.525231 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:39.525236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:39.525239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:39.525736 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:39.526021 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:39.526350 (+ 329us) server_negotiation.cc:299] Negotiation successful
0618 22:38:39.526540 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17784,"thread_start_us":17687,"threads_started":1}
W20260618 22:38:39.527693 7300 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44318
I20260618 22:38:39.547073 7639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:39.531341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36504 (local address 127.5.213.254:34919)
0618 22:38:39.531538 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:39.531545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:39.531976 (+ 431us) server_negotiation.cc:407] Connection header received
0618 22:38:39.532164 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:39.532170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:39.532242 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:39.532306 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:39.533954 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.535081 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.536879 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.537619 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.537765 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:39.540664 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:39.540670 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:39.540675 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:39.540715 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:39.544115 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:39.544866 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:39.544871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:39.544875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:39.545013 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:39.545448 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:39.545453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:39.545457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:39.545783 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:39.545977 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:39.546543 (+ 566us) server_negotiation.cc:299] Negotiation successful
0618 22:38:39.546779 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260618 22:38:39.548460 7097 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36504
I20260618 22:38:39.550277 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 7169
I20260618 22:38:39.564386 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:39.552268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.5.213.193:44015)
0618 22:38:39.552434 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:39.552439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:39.552526 (+ 87us) server_negotiation.cc:407] Connection header received
0618 22:38:39.552769 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:39.552774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:39.552868 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:39.552931 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:39.554484 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.555235 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.556917 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.557341 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.557456 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:39.559652 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:39.559659 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:39.559662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:39.559698 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:39.562084 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:39.562685 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:39.562699 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:39.562703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:39.562827 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:39.563152 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:39.563157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:39.563161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:39.563406 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:39.563626 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:39.563884 (+ 258us) server_negotiation.cc:299] Negotiation successful
0618 22:38:39.564085 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260618 22:38:39.564903 7300 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44332
W20260618 22:38:39.565387 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 7169: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:38:39.565552 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 7169
I20260618 22:38:39.578886 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 7309
Jun 18 22:38:39 dist-test-slave-lnj3 krb5kdc[7053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822318, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.213.194@KRBTEST.COM
I20260618 22:38:39.595381 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:39.579721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34622 (local address 127.5.213.194:40375)
0618 22:38:39.579914 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:39.579920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:39.579944 (+ 24us) server_negotiation.cc:407] Connection header received
0618 22:38:39.580006 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:39.580010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:39.580082 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:39.580193 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:39.581248 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.582409 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.583640 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.584273 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.584413 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:39.588460 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:39.588479 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:39.588501 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:39.588549 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:39.592495 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:39.593217 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:39.593231 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:39.593235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:39.593429 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:39.593781 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:39.593786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:39.593793 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:39.594240 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:39.594559 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:39.594763 (+ 204us) server_negotiation.cc:299] Negotiation successful
0618 22:38:39.594975 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260618 22:38:39.596113 7439 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34622
W20260618 22:38:39.596756 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 7309: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:38:39.596895 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 7309
I20260618 22:38:39.614830 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 7448
Jun 18 22:38:39 dist-test-slave-lnj3 krb5kdc[7053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822318, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.213.195@KRBTEST.COM
I20260618 22:38:39.628762 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:39.615486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35128 (local address 127.5.213.195:45381)
0618 22:38:39.615898 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:39.615902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:39.615917 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:38:39.615972 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:39.615975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:39.616039 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:39.616127 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:39.617243 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.617924 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.619148 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:39.619587 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:39.619682 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:39.623411 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:39.623425 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:39.623441 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:39.623475 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:39.626313 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:39.627107 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:39.627114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:39.627116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:39.627236 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:39.627565 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:39.627569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:39.627571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:39.627900 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:39.628115 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:39.628307 (+ 192us) server_negotiation.cc:299] Negotiation successful
0618 22:38:39.628461 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":223,"threads_started":1}
W20260618 22:38:39.629364 7578 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35128
W20260618 22:38:39.629999 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 7448: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:38:39.630137 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 7448
I20260618 22:38:39.642939 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 7066
I20260618 22:38:39.643980 7138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:38:39.752851 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 7066
2026-06-18T22:38:39Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4349 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:38:39 dist-test-slave-lnj3 krb5kdc[7677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:38:39 dist-test-slave-lnj3 krb5kdc[7677](info): set up 2 sockets
Jun 18 22:38:39 dist-test-slave-lnj3 krb5kdc[7677](info): commencing operation
krb5kdc: starting...
W20260618 22:38:40.734354 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:38:40 dist-test-slave-lnj3 krb5kdc[7677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:38:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:38:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:38:40.891876 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:33739
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:33739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:41.197458 7690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:41.197752 7690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:41.197863 7690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:41.204957 7690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:38:41.205052 7690 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260618 22:38:41.205111 7690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:41.205192 7690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:41.205247 7690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:38:41.205343 7690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:38:41.215019 7690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:33739
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:33739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:41.216408 7690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:41.217738 7690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:41.229071 7695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:41.229065 7698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:41.229106 7696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:41.229995 7690 server_base.cc:1034] running on GCE node
I20260618 22:38:41.230854 7690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:41.232259 7690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:41.233489 7690 hybrid_clock.cc:648] HybridClock initialized: now 1781822321233421 us; error 39 us; skew 500 ppm
I20260618 22:38:41.233832 7690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:41 dist-test-slave-lnj3 krb5kdc[7677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822321, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:41.240919 7690 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:38:41.242238 7690 webserver.cc:460] Webserver started at http://127.5.213.254:46681/ using document root <none> and password file <none>
I20260618 22:38:41.242935 7690 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:41.243031 7690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:41.243319 7690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:41.246109 7690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "d0f3aeca28c8467dadf0ea56af4c1f6d"
format_stamp: "Formatted at 2026-06-18 22:38:41 on dist-test-slave-lnj3"
server_key: "ce4ef47ba243f495bf90f2859ed37b75"
server_key_iv: "7ba0e5612a59e09430068a47c8d07a8d"
server_key_version: "encryptionkey@0"
I20260618 22:38:41.246869 7690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "d0f3aeca28c8467dadf0ea56af4c1f6d"
format_stamp: "Formatted at 2026-06-18 22:38:41 on dist-test-slave-lnj3"
server_key: "ce4ef47ba243f495bf90f2859ed37b75"
server_key_iv: "7ba0e5612a59e09430068a47c8d07a8d"
server_key_version: "encryptionkey@0"
I20260618 22:38:41.251645 7690 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260618 22:38:41.255137 7705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:41.256433 7690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260618 22:38:41.256580 7690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "d0f3aeca28c8467dadf0ea56af4c1f6d"
format_stamp: "Formatted at 2026-06-18 22:38:41 on dist-test-slave-lnj3"
server_key: "ce4ef47ba243f495bf90f2859ed37b75"
server_key_iv: "7ba0e5612a59e09430068a47c8d07a8d"
server_key_version: "encryptionkey@0"
I20260618 22:38:41.256770 7690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:41.269016 7690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:41.272432 7690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:41.272648 7690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:41.291139 7690 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:33739
I20260618 22:38:41.291157 7766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:33739 every 8 connection(s)
I20260618 22:38:41.292361 7690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:38:41.295341 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 7690
I20260618 22:38:41.295502 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:38:41.295915 7767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:41.296113 5975 external_mini_cluster.cc:1428] Setting key e464de518869debf95bad8afb4f9515f
I20260618 22:38:41.309345 7767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d: Bootstrap starting.
Jun 18 22:38:41 dist-test-slave-lnj3 krb5kdc[7677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:41.312930 7767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:41.314173 7767 log.cc:824] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d: Log is configured to *not* fsync() on all Append() calls
I20260618 22:38:41.316944 7767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d: No bootstrap required, opened a new log
I20260618 22:38:41.318137 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:41.297882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.5.213.254:33739)
0618 22:38:41.298686 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:41.298704 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:41.298751 (+ 47us) server_negotiation.cc:407] Connection header received
0618 22:38:41.300495 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:41.300520 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:41.301015 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:41.301436 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:41.302583 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:41.304692 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:41.305886 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:41.306970 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:41.307159 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:41.310797 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:41.310812 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:41.310831 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:41.310882 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:41.314335 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:41.314977 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:41.314983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:41.314988 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:41.315152 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:41.315478 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:41.315482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:41.315484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:41.315965 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:41.316201 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:41.316763 (+ 562us) server_negotiation.cc:299] Negotiation successful
0618 22:38:41.317179 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":316,"threads_started":1}
I20260618 22:38:41.324400 7767 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f3aeca28c8467dadf0ea56af4c1f6d" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33739 } }
I20260618 22:38:41.324714 7767 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:41.324803 7767 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f3aeca28c8467dadf0ea56af4c1f6d, State: Initialized, Role: FOLLOWER
I20260618 22:38:41.325281 7767 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f3aeca28c8467dadf0ea56af4c1f6d" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33739 } }
I20260618 22:38:41.325474 7767 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:38:41.325584 7767 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:38:41.325728 7767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:41.328011 7767 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f3aeca28c8467dadf0ea56af4c1f6d" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33739 } }
I20260618 22:38:41.328441 7767 leader_election.cc:302] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0f3aeca28c8467dadf0ea56af4c1f6d; no voters:
I20260618 22:38:41.328927 7767 leader_election.cc:288] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:38:41.329030 7772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:38:41.329631 7772 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d [term 1 LEADER]: Becoming Leader. State: Replica: d0f3aeca28c8467dadf0ea56af4c1f6d, State: Running, Role: LEADER
I20260618 22:38:41.330090 7772 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f3aeca28c8467dadf0ea56af4c1f6d" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33739 } }
I20260618 22:38:41.330693 7767 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:38:41.332515 7774 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0f3aeca28c8467dadf0ea56af4c1f6d. Latest consensus state: current_term: 1 leader_uuid: "d0f3aeca28c8467dadf0ea56af4c1f6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f3aeca28c8467dadf0ea56af4c1f6d" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33739 } } }
I20260618 22:38:41.332897 7774 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d [sys.catalog]: This master's current role is: LEADER
I20260618 22:38:41.332525 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0f3aeca28c8467dadf0ea56af4c1f6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f3aeca28c8467dadf0ea56af4c1f6d" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33739 } } }
I20260618 22:38:41.333621 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d [sys.catalog]: This master's current role is: LEADER
I20260618 22:38:41.335659 7779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:38:41.340603 7779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:38:41.348843 7779 catalog_manager.cc:1347] Generated new cluster ID: e35d22f7a1be4c3898c9e0d3b9366100
I20260618 22:38:41.348945 7779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:38:41.366380 7779 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:38:41.367609 7779 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:38:41.374449 7779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d: Generated new TSK 0
I20260618 22:38:41.375187 7779 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
I20260618 22:38:41.418109 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:33739
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:41.725533 7793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:41.725853 7793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:41.725987 7793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:41.733264 7793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:41.733385 7793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:41.733508 7793 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260618 22:38:41.733572 7793 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:38:41.733673 7793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:38:41.743485 7793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.213.254:33739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:41.744715 7793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:41.746011 7793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:41.758638 7798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:41.758662 7799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:41.758670 7801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:41.759418 7793 server_base.cc:1034] running on GCE node
I20260618 22:38:41.760068 7793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:41.761004 7793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:41.762194 7793 hybrid_clock.cc:648] HybridClock initialized: now 1781822321762136 us; error 51 us; skew 500 ppm
I20260618 22:38:41.762527 7793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:41 dist-test-slave-lnj3 krb5kdc[7677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822321, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:41.765453 7793 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:38:41.766775 7793 webserver.cc:460] Webserver started at http://127.5.213.193:43151/ using document root <none> and password file <none>
I20260618 22:38:41.767419 7793 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:41.767529 7793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:41.767817 7793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:41.770524 7793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "7e3989d8c725426da93f7200472bce7b"
format_stamp: "Formatted at 2026-06-18 22:38:41 on dist-test-slave-lnj3"
server_key: "06b26e050e35d5e364ce87779e606e0f"
server_key_iv: "e1beb1114ef6934daad52248d160013d"
server_key_version: "encryptionkey@0"
I20260618 22:38:41.771261 7793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "7e3989d8c725426da93f7200472bce7b"
format_stamp: "Formatted at 2026-06-18 22:38:41 on dist-test-slave-lnj3"
server_key: "06b26e050e35d5e364ce87779e606e0f"
server_key_iv: "e1beb1114ef6934daad52248d160013d"
server_key_version: "encryptionkey@0"
I20260618 22:38:41.776072 7793 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260618 22:38:41.779294 7808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:41.780544 7793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:38:41.780685 7793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "7e3989d8c725426da93f7200472bce7b"
format_stamp: "Formatted at 2026-06-18 22:38:41 on dist-test-slave-lnj3"
server_key: "06b26e050e35d5e364ce87779e606e0f"
server_key_iv: "e1beb1114ef6934daad52248d160013d"
server_key_version: "encryptionkey@0"
I20260618 22:38:41.780845 7793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:41.796232 7793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:41.799875 7793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:41.800127 7793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:41.802567 7793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:41.802699 7793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:41.802809 7793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:41.802891 7793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:38:41 dist-test-slave-lnj3 krb5kdc[7677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822321, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:41.824997 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:41.803314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:58885 (local address 127.5.213.254:33739)
0618 22:38:41.803549 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:41.803555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:41.804854 (+ 1299us) server_negotiation.cc:407] Connection header received
0618 22:38:41.807086 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:41.807092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:41.807176 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:41.807252 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:41.809626 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:41.810579 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:41.812915 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:41.813558 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:41.813711 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:41.818119 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:41.818124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:41.818128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:41.818162 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:41.821099 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:41.822124 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:41.822128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:41.822130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:41.822256 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:41.822860 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:41.822865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:41.822869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:41.823215 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:41.823349 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:41.824429 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0618 22:38:41.824681 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260618 22:38:41.826551 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:41.803917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33739 (local address 127.5.213.193:58885)
0618 22:38:41.804688 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:41.804736 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:41.806737 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:41.807751 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:41.807762 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:41.808550 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:41.809454 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:41.809486 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:41.810759 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:41.810764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:41.812742 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:41.812756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:41.814009 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:41.814020 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:41.814260 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:41.815028 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:41.815051 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:41.817859 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:41.821239 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:41.821253 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:41.821280 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:41.821959 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:41.822382 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:41.822386 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:41.822393 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:41.822728 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:41.823344 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:41.823356 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:41.823946 (+ 590us) client_negotiation.cc:769] Sending connection context
0618 22:38:41.824316 (+ 370us) client_negotiation.cc:240] Negotiation successful
0618 22:38:41.824950 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":329,"threads_started":1}
I20260618 22:38:41.841445 7793 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:33271
I20260618 22:38:41.841514 7927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:33271 every 8 connection(s)
I20260618 22:38:41.843119 7793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:38:41.844491 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 7793
I20260618 22:38:41.844673 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:38:41.845278 7928 heartbeater.cc:347] Connected to a master server at 127.5.213.254:33739
I20260618 22:38:41.845402 5975 external_mini_cluster.cc:1428] Setting key 2c98442f241fffc94ee4ad5db44a4425
I20260618 22:38:41.845497 7928 heartbeater.cc:464] Registering TS with master...
I20260618 22:38:41.846282 7928 heartbeater.cc:511] Master 127.5.213.254:33739 requested a full tablet report, sending...
I20260618 22:38:41.848578 7722 ts_manager.cc:194] Registered new tserver with Master: 7e3989d8c725426da93f7200472bce7b (127.5.213.193:33271)
I20260618 22:38:41.849074 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:41.833617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:52663 (local address 127.5.213.254:33739)
0618 22:38:41.833771 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:41.833775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:41.834884 (+ 1109us) server_negotiation.cc:407] Connection header received
0618 22:38:41.835127 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:41.835133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:41.835214 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:41.835285 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:41.836873 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:41.837936 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:41.839918 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:41.840448 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:41.840572 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:41.843054 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:41.843059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:41.843062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:41.843093 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:41.846190 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:41.847014 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:41.847019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:41.847023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:41.847180 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:41.847550 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:41.847554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:41.847557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:41.847925 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:41.848063 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:41.848521 (+ 458us) server_negotiation.cc:299] Negotiation successful
0618 22:38:41.848756 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 22:38:41.849087 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:41.833626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33739 (local address 127.5.213.193:52663)
0618 22:38:41.834763 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:41.834785 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:41.834947 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:41.835507 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:41.835511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:41.836216 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:41.836677 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:41.836695 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:41.838129 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:41.838137 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:41.839796 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:41.839807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:41.840713 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:41.840720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:41.840944 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:41.841894 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:41.841915 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:41.842903 (+ 988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:41.846329 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:41.846334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:41.846338 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:41.846874 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:41.847313 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:41.847319 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:41.847323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:41.847446 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:41.848050 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:41.848054 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:41.848244 (+ 190us) client_negotiation.cc:769] Sending connection context
0618 22:38:41.848379 (+ 135us) client_negotiation.cc:240] Negotiation successful
0618 22:38:41.848674 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"thread_start_us":490,"threads_started":1}
I20260618 22:38:41.850260 7722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:58885
WARNING: no policy specified for kudu/127.5.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
I20260618 22:38:41.881464 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:33739
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:42.183162 7932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:42.183421 7932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:42.183553 7932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:42.190657 7932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:42.190775 7932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:42.190899 7932 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260618 22:38:42.190958 7932 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:38:42.191056 7932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:38:42.200505 7932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.213.254:33739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:42.201684 7932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:42.202941 7932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:42.214504 7940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:42.214529 7938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:42.214735 7937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:42.215245 7932 server_base.cc:1034] running on GCE node
I20260618 22:38:42.215811 7932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:42.216627 7932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:42.217820 7932 hybrid_clock.cc:648] HybridClock initialized: now 1781822322217777 us; error 35 us; skew 500 ppm
I20260618 22:38:42.218092 7932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:42 dist-test-slave-lnj3 krb5kdc[7677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822322, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:42.220980 7932 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:38:42.222294 7932 webserver.cc:460] Webserver started at http://127.5.213.194:34523/ using document root <none> and password file <none>
I20260618 22:38:42.223066 7932 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:42.223186 7932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:42.223466 7932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:42.226306 7932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "0a7f01d4d06f4b7eba5cb53475417fd6"
format_stamp: "Formatted at 2026-06-18 22:38:42 on dist-test-slave-lnj3"
server_key: "84734e592c6ad11c319ec4a4e029ce20"
server_key_iv: "003ba5157ccd0dba0503cb77c1421148"
server_key_version: "encryptionkey@0"
I20260618 22:38:42.227015 7932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "0a7f01d4d06f4b7eba5cb53475417fd6"
format_stamp: "Formatted at 2026-06-18 22:38:42 on dist-test-slave-lnj3"
server_key: "84734e592c6ad11c319ec4a4e029ce20"
server_key_iv: "003ba5157ccd0dba0503cb77c1421148"
server_key_version: "encryptionkey@0"
I20260618 22:38:42.231986 7932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260618 22:38:42.235196 7947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:42.236483 7932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:38:42.236622 7932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "0a7f01d4d06f4b7eba5cb53475417fd6"
format_stamp: "Formatted at 2026-06-18 22:38:42 on dist-test-slave-lnj3"
server_key: "84734e592c6ad11c319ec4a4e029ce20"
server_key_iv: "003ba5157ccd0dba0503cb77c1421148"
server_key_version: "encryptionkey@0"
I20260618 22:38:42.236802 7932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:42.256623 7932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:42.260353 7932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:42.260597 7932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:42.263103 7932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:42.263201 7932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:42.263307 7932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:42.263379 7932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:38:42 dist-test-slave-lnj3 krb5kdc[7677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822322, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:42.284317 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:42.263617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:58801 (local address 127.5.213.254:33739)
0618 22:38:42.263765 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:42.263769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:42.265182 (+ 1413us) server_negotiation.cc:407] Connection header received
0618 22:38:42.267909 (+ 2727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:42.267914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:42.267992 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:42.268084 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:42.270105 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.270961 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.273765 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.274411 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.274539 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:42.279027 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:42.279033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:42.279037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:42.279069 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:42.281363 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:42.281989 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:42.281995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:42.281999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:42.282115 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:42.282492 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:42.282497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:42.282501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:42.282804 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:42.282936 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:42.283746 (+ 810us) server_negotiation.cc:299] Negotiation successful
0618 22:38:42.283995 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:38:42.285693 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:42.264204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33739 (local address 127.5.213.194:58801)
0618 22:38:42.264995 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:42.265054 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:42.267587 (+ 2533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:42.268391 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:42.268405 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:42.269121 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:42.269927 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:42.269959 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.271154 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.271170 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:42.273565 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:42.273584 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.274701 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.274710 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:42.274985 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:42.275732 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:42.275757 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:42.278815 (+ 3058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:42.281484 (+ 2669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:42.281493 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:42.281510 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:42.281889 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:42.282212 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:42.282218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:42.282222 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:42.282397 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:42.282922 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:42.282932 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:42.283333 (+ 401us) client_negotiation.cc:769] Sending connection context
0618 22:38:42.283679 (+ 346us) client_negotiation.cc:240] Negotiation successful
0618 22:38:42.284225 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":298,"threads_started":1}
I20260618 22:38:42.308974 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:42.294821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:34551 (local address 127.5.213.254:33739)
0618 22:38:42.294989 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:42.294994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:42.295011 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:38:42.295059 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:42.295063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:42.295117 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:42.295210 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:42.296314 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.297593 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.299542 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.300256 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.300382 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:42.302490 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:42.302497 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:42.302503 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:42.302542 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:42.306172 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:42.307108 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:42.307115 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:42.307119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:42.307284 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:42.307681 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:42.307685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:42.307687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:42.307956 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:42.308063 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:42.308561 (+ 498us) server_negotiation.cc:299] Negotiation successful
0618 22:38:42.308764 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 22:38:42.309414 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:42.291274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33739 (local address 127.5.213.194:34551)
0618 22:38:42.291899 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:42.291921 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:42.292038 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:42.295251 (+ 3213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:42.295256 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:42.295798 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:42.296132 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:42.296144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.297738 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.297743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:42.299396 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:42.299410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.300442 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.300449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:42.300626 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:42.301326 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:42.301349 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:42.302298 (+ 949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:42.306334 (+ 4036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:42.306339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:42.306343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:42.306953 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:42.307412 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:42.307417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:42.307421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:42.307570 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:42.308053 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:42.308058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:42.308284 (+ 226us) client_negotiation.cc:769] Sending connection context
0618 22:38:42.308451 (+ 167us) client_negotiation.cc:240] Negotiation successful
0618 22:38:42.308793 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":320,"threads_started":1}
I20260618 22:38:42.322016 7932 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:37157
I20260618 22:38:42.322046 8066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:37157 every 8 connection(s)
I20260618 22:38:42.323206 7932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:38:42.325017 8067 heartbeater.cc:347] Connected to a master server at 127.5.213.254:33739
I20260618 22:38:42.325207 8067 heartbeater.cc:464] Registering TS with master...
I20260618 22:38:42.325788 8067 heartbeater.cc:511] Master 127.5.213.254:33739 requested a full tablet report, sending...
I20260618 22:38:42.326812 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 7932
I20260618 22:38:42.326973 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:38:42.327051 7722 ts_manager.cc:194] Registered new tserver with Master: 0a7f01d4d06f4b7eba5cb53475417fd6 (127.5.213.194:37157)
I20260618 22:38:42.327502 5975 external_mini_cluster.cc:1428] Setting key ae5964730640fb361bb4ee8eca03e40a
I20260618 22:38:42.328095 7722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:58801
WARNING: no policy specified for kudu/127.5.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
I20260618 22:38:42.361821 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:33739
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:42.664652 8071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:42.664938 8071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:42.665084 8071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:42.672091 8071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:42.672219 8071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:42.672346 8071 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260618 22:38:42.672412 8071 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:38:42.672514 8071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:38:42.682087 8071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.213.254:33739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:42.683355 8071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:42.684686 8071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:42.696837 8077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:42.696877 8076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:42.696903 8079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:42.697537 8071 server_base.cc:1034] running on GCE node
I20260618 22:38:42.698079 8071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:42.698984 8071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:42.700165 8071 hybrid_clock.cc:648] HybridClock initialized: now 1781822322700128 us; error 31 us; skew 500 ppm
I20260618 22:38:42.700426 8071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:42 dist-test-slave-lnj3 krb5kdc[7677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822322, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:42.703207 8071 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:38:42.704521 8071 webserver.cc:460] Webserver started at http://127.5.213.195:35087/ using document root <none> and password file <none>
I20260618 22:38:42.705144 8071 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:42.705233 8071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:42.705468 8071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:42.708163 8071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "eebd71fa48eb46cd8d0f506ffbff04e0"
format_stamp: "Formatted at 2026-06-18 22:38:42 on dist-test-slave-lnj3"
server_key: "7cd18189bbabe5e73a09e35cba2c6ab2"
server_key_iv: "14154debed9cc01306e99b8d71555d57"
server_key_version: "encryptionkey@0"
I20260618 22:38:42.708822 8071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "eebd71fa48eb46cd8d0f506ffbff04e0"
format_stamp: "Formatted at 2026-06-18 22:38:42 on dist-test-slave-lnj3"
server_key: "7cd18189bbabe5e73a09e35cba2c6ab2"
server_key_iv: "14154debed9cc01306e99b8d71555d57"
server_key_version: "encryptionkey@0"
I20260618 22:38:42.713586 8071 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260618 22:38:42.717087 8086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:42.718318 8071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260618 22:38:42.718437 8071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "eebd71fa48eb46cd8d0f506ffbff04e0"
format_stamp: "Formatted at 2026-06-18 22:38:42 on dist-test-slave-lnj3"
server_key: "7cd18189bbabe5e73a09e35cba2c6ab2"
server_key_iv: "14154debed9cc01306e99b8d71555d57"
server_key_version: "encryptionkey@0"
I20260618 22:38:42.718581 8071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:42.734850 8071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:42.738806 8071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:42.739077 8071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:42.741530 8071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:42.741626 8071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:42.741735 8071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:42.741827 8071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:38:42 dist-test-slave-lnj3 krb5kdc[7677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822322, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:42.763584 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:42.741977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:34233 (local address 127.5.213.254:33739)
0618 22:38:42.742180 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:42.742187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:42.743402 (+ 1215us) server_negotiation.cc:407] Connection header received
0618 22:38:42.745479 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:42.745485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:42.745574 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:42.745658 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:42.747342 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.748317 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.751162 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.751604 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.751700 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:42.756939 (+ 5239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:42.756946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:42.756951 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:42.756994 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:42.760376 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:42.761279 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:42.761285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:42.761289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:42.761459 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:42.761837 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:42.761841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:42.761845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:42.762178 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:42.762333 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:42.763028 (+ 695us) server_negotiation.cc:299] Negotiation successful
0618 22:38:42.763242 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260618 22:38:42.764650 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:42.742484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33739 (local address 127.5.213.195:34233)
0618 22:38:42.743256 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:42.743306 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:42.745191 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:42.745918 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:42.745926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:42.746409 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:42.747182 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:42.747206 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.748527 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.748533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:42.751007 (+ 2474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:42.751030 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.751831 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.751842 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:42.752155 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:42.752953 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:42.752983 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:42.756623 (+ 3640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:42.760552 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:42.760564 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:42.760584 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:42.761141 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:42.761557 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:42.761560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:42.761563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:42.761742 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:42.762278 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:42.762284 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:42.762617 (+ 333us) client_negotiation.cc:769] Sending connection context
0618 22:38:42.762927 (+ 310us) client_negotiation.cc:240] Negotiation successful
0618 22:38:42.763451 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":345,"threads_started":1}
I20260618 22:38:42.783277 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:42.770055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:35533 (local address 127.5.213.254:33739)
0618 22:38:42.770216 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:42.770233 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:42.770827 (+ 594us) server_negotiation.cc:407] Connection header received
0618 22:38:42.770986 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:42.770989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:42.771046 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:42.771121 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:42.772213 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.772905 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.774854 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.775345 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.776040 (+ 695us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:42.777381 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:42.777388 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:42.777393 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:42.777426 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:42.780474 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:42.781273 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:42.781278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:42.781282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:42.781439 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:42.781835 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:42.781840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:42.781843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:42.782162 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:42.782295 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:42.782736 (+ 441us) server_negotiation.cc:299] Negotiation successful
0618 22:38:42.782928 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 22:38:42.783381 8158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:42.770093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33739 (local address 127.5.213.195:35533)
0618 22:38:42.770734 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:42.770753 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:42.770887 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:42.771163 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:42.771167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:42.771730 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:42.772094 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:42.772105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.773005 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.773011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:42.774595 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:42.774608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.775554 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.775558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:42.775737 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:42.776371 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:42.776393 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:42.777208 (+ 815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:42.780640 (+ 3432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:42.780644 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:42.780648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:42.781169 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:42.781571 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:42.781576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:42.781580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:42.781742 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:42.782258 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:42.782263 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:42.782453 (+ 190us) client_negotiation.cc:769] Sending connection context
0618 22:38:42.782589 (+ 136us) client_negotiation.cc:240] Negotiation successful
0618 22:38:42.782936 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":314,"threads_started":1}
I20260618 22:38:42.792486 8071 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:43811
I20260618 22:38:42.792512 8205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:43811 every 8 connection(s)
I20260618 22:38:42.793673 8071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:38:42.795478 8206 heartbeater.cc:347] Connected to a master server at 127.5.213.254:33739
I20260618 22:38:42.795688 8206 heartbeater.cc:464] Registering TS with master...
I20260618 22:38:42.796403 8206 heartbeater.cc:511] Master 127.5.213.254:33739 requested a full tablet report, sending...
I20260618 22:38:42.796437 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 8071
I20260618 22:38:42.796567 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:38:42.796972 5975 external_mini_cluster.cc:1428] Setting key 56fbaba39181cfcd1023c97690064098
I20260618 22:38:42.797729 7721 ts_manager.cc:194] Registered new tserver with Master: eebd71fa48eb46cd8d0f506ffbff04e0 (127.5.213.195:43811)
I20260618 22:38:42.798760 7721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:34233
I20260618 22:38:42.799633 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 22:38:42 dist-test-slave-lnj3 krb5kdc[7677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 22:38:42 dist-test-slave-lnj3 krb5kdc[7677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:42.840107 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:42.827839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.5.213.254:33739)
0618 22:38:42.827993 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:42.827997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:42.828312 (+ 315us) server_negotiation.cc:407] Connection header received
0618 22:38:42.828460 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:42.828466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:42.828535 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:42.828620 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:42.829906 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.830737 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.831894 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.832298 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.832399 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:42.835802 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:42.835807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:42.835810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:42.835842 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:42.838000 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:42.838526 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:42.838530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:42.838532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:42.838711 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:42.838994 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:42.838996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:42.838998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:42.839243 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:42.839409 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:42.839635 (+ 226us) server_negotiation.cc:299] Negotiation successful
0618 22:38:42.839811 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:38:42.844058 7722 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:38:42.850494 7722 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:38:42.853892 7928 heartbeater.cc:503] Master 127.5.213.254:33739 was elected leader, sending a full tablet report...
I20260618 22:38:42.882700 8223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:42.862068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54686 (local address 127.5.213.194:37157)
0618 22:38:42.875515 (+ 13447us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:42.875524 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:42.876258 (+ 734us) server_negotiation.cc:407] Connection header received
0618 22:38:42.876382 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:42.876390 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:42.876652 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:42.876846 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:38:42.877498 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.878553 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.880812 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.881675 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.881792 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:42.882009 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:42.882232 (+ 223us) server_negotiation.cc:299] Negotiation successful
0618 22:38:42.882389 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13306,"thread_start_us":13194,"threads_started":1}
I20260618 22:38:42.882758 8222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:42.861386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:37157 (local address 127.0.0.1:54686)
0618 22:38:42.875992 (+ 14606us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:42.876011 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:42.876133 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:42.876863 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:42.876870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:42.876895 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:38:42.877301 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:42.877312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.878702 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.878707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:42.880676 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:42.880687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.881854 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.881860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:42.882032 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:42.882075 (+ 43us) client_negotiation.cc:769] Sending connection context
0618 22:38:42.882186 (+ 111us) client_negotiation.cc:240] Negotiation successful
0618 22:38:42.882369 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14464,"mutex_wait_us":11,"thread_start_us":14936,"threads_started":1}
I20260618 22:38:42.884230 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:42.861708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:43811 (local address 127.0.0.1:59582)
0618 22:38:42.876547 (+ 14839us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:42.876562 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:42.876681 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:42.877198 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:42.877206 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:42.877230 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:38:42.877680 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:42.877692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.879544 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.879549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:42.881742 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:42.881751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.883534 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.883540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:42.883704 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:42.883734 (+ 30us) client_negotiation.cc:769] Sending connection context
0618 22:38:42.883814 (+ 80us) client_negotiation.cc:240] Negotiation successful
0618 22:38:42.883927 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14740,"mutex_wait_us":8,"thread_start_us":13996,"threads_started":1}
I20260618 22:38:42.884296 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:42.862499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59582 (local address 127.5.213.195:43811)
0618 22:38:42.875144 (+ 12645us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:42.875153 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:42.876669 (+ 1516us) server_negotiation.cc:407] Connection header received
0618 22:38:42.876809 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:42.876816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:42.877006 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:42.877180 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:38:42.877873 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.879406 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.881905 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.883341 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.883462 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:42.883678 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:42.883865 (+ 187us) server_negotiation.cc:299] Negotiation successful
0618 22:38:42.884005 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12476,"thread_start_us":12316,"threads_started":1}
I20260618 22:38:42.885330 8219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:42.861224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:33271 (local address 127.0.0.1:46294)
0618 22:38:42.873587 (+ 12363us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:42.873642 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:42.873828 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:42.877068 (+ 3240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:42.877072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:42.877097 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:38:42.877573 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:42.877583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.879395 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.879400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:42.881696 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:42.881709 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.884580 (+ 2871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.884585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:42.884715 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:42.884762 (+ 47us) client_negotiation.cc:769] Sending connection context
0618 22:38:42.884842 (+ 80us) client_negotiation.cc:240] Negotiation successful
0618 22:38:42.884957 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12084,"mutex_wait_us":50,"spinlock_wait_cycles":1792,"thread_start_us":12789,"threads_started":1}
I20260618 22:38:42.887140 7997 tablet_service.cc:1467] Processing CreateTablet for tablet 1793de429e384257ae03bc0e23b7da02 (DEFAULT_TABLE table=test-table [id=da14333fed4449159e08296adeef01c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:42.887763 8136 tablet_service.cc:1467] Processing CreateTablet for tablet 1793de429e384257ae03bc0e23b7da02 (DEFAULT_TABLE table=test-table [id=da14333fed4449159e08296adeef01c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:42.888689 7997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1793de429e384257ae03bc0e23b7da02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:42.889146 8136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1793de429e384257ae03bc0e23b7da02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:42.892692 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:42.861402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46294 (local address 127.5.213.193:33271)
0618 22:38:42.876295 (+ 14893us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:42.876304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:42.876335 (+ 31us) server_negotiation.cc:407] Connection header received
0618 22:38:42.876430 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:42.876438 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:42.876682 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:42.876863 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:38:42.877733 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.879232 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.881883 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:42.883344 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:42.883461 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:42.883649 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:42.892191 (+ 8542us) server_negotiation.cc:299] Negotiation successful
0618 22:38:42.892389 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14724,"thread_start_us":14607,"threads_started":1}
I20260618 22:38:42.895565 7858 tablet_service.cc:1467] Processing CreateTablet for tablet 1793de429e384257ae03bc0e23b7da02 (DEFAULT_TABLE table=test-table [id=da14333fed4449159e08296adeef01c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:42.896621 7858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1793de429e384257ae03bc0e23b7da02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:42.903345 8226 tablet_bootstrap.cc:492] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0: Bootstrap starting.
I20260618 22:38:42.906841 8226 tablet_bootstrap.cc:654] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:42.907618 8225 tablet_bootstrap.cc:492] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6: Bootstrap starting.
I20260618 22:38:42.908120 8226 log.cc:824] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0: Log is configured to *not* fsync() on all Append() calls
I20260618 22:38:42.908973 8227 tablet_bootstrap.cc:492] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b: Bootstrap starting.
I20260618 22:38:42.911026 8225 tablet_bootstrap.cc:654] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:42.911618 8226 tablet_bootstrap.cc:492] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0: No bootstrap required, opened a new log
I20260618 22:38:42.911902 8226 ts_tablet_manager.cc:1397] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260618 22:38:42.912154 8227 tablet_bootstrap.cc:654] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:42.912333 8225 log.cc:824] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6: Log is configured to *not* fsync() on all Append() calls
I20260618 22:38:42.913200 8227 log.cc:824] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:38:42.915005 8225 tablet_bootstrap.cc:492] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6: No bootstrap required, opened a new log
I20260618 22:38:42.915266 8225 ts_tablet_manager.cc:1397] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260618 22:38:42.915899 8227 tablet_bootstrap.cc:492] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b: No bootstrap required, opened a new log
I20260618 22:38:42.916134 8227 ts_tablet_manager.cc:1397] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260618 22:38:42.919098 8226 raft_consensus.cc:348] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:42.919425 8226 raft_consensus.cc:374] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:42.919530 8226 raft_consensus.cc:729] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebd71fa48eb46cd8d0f506ffbff04e0, State: Initialized, Role: FOLLOWER
I20260618 22:38:42.920116 8226 consensus_queue.cc:260] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:42.921165 8206 heartbeater.cc:503] Master 127.5.213.254:33739 was elected leader, sending a full tablet report...
I20260618 22:38:42.921766 8226 ts_tablet_manager.cc:1428] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260618 22:38:42.925828 8225 raft_consensus.cc:348] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:42.926272 8225 raft_consensus.cc:374] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:42.926369 8225 raft_consensus.cc:729] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a7f01d4d06f4b7eba5cb53475417fd6, State: Initialized, Role: FOLLOWER
I20260618 22:38:42.926236 8227 raft_consensus.cc:348] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:42.926728 8227 raft_consensus.cc:374] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:42.926826 8227 raft_consensus.cc:729] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e3989d8c725426da93f7200472bce7b, State: Initialized, Role: FOLLOWER
I20260618 22:38:42.927104 8225 consensus_queue.cc:260] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:42.927369 8227 consensus_queue.cc:260] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:42.928170 8067 heartbeater.cc:503] Master 127.5.213.254:33739 was elected leader, sending a full tablet report...
I20260618 22:38:42.928833 8225 ts_tablet_manager.cc:1428] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260618 22:38:42.928815 8227 ts_tablet_manager.cc:1428] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260618 22:38:43.045135 8207 tablet.cc:2369] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:38:43.059690 8232 raft_consensus.cc:482] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:38:43.060007 8232 raft_consensus.cc:504] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.061756 8232 leader_election.cc:288] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e3989d8c725426da93f7200472bce7b (127.5.213.193:33271), eebd71fa48eb46cd8d0f506ffbff04e0 (127.5.213.195:43811)
I20260618 22:38:43.070297 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.062159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:33589 (local address 127.5.213.195:43811)
0618 22:38:43.062351 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:43.062357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:43.063015 (+ 658us) server_negotiation.cc:407] Connection header received
0618 22:38:43.063210 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:43.063216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:43.063296 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:43.063399 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:38:43.064140 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.065248 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.068103 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.069272 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.069431 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.069504 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:43.069931 (+ 427us) server_negotiation.cc:299] Negotiation successful
0618 22:38:43.070053 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:38:43.070314 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.062159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:43807 (local address 127.5.213.193:33271)
0618 22:38:43.062356 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:43.062362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:43.063091 (+ 729us) server_negotiation.cc:407] Connection header received
0618 22:38:43.063213 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:43.063218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:43.063309 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:43.063392 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:38:43.064140 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.065268 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.068103 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.069293 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.069431 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.069504 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:43.069931 (+ 427us) server_negotiation.cc:299] Negotiation successful
0618 22:38:43.070053 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260618 22:38:43.070511 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.062081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:43811 (local address 127.5.213.194:33589)
0618 22:38:43.062907 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:43.062926 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:43.063078 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:43.063416 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:43.063421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:43.063444 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:38:43.063979 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.063996 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.065372 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.065378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.067971 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.067979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.069502 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.069507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.069721 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.069814 (+ 93us) client_negotiation.cc:769] Sending connection context
0618 22:38:43.069901 (+ 87us) client_negotiation.cc:240] Negotiation successful
0618 22:38:43.070061 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":244,"thread_start_us":472,"threads_started":1}
I20260618 22:38:43.070534 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.062156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:33271 (local address 127.5.213.194:43807)
0618 22:38:43.062998 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:43.063011 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:43.063109 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:43.063421 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:43.063425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:43.063470 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:38:43.064008 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.064016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.065373 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.065378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.067963 (+ 2585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.067977 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.069487 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.069494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.069724 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.069812 (+ 88us) client_negotiation.cc:769] Sending connection context
0618 22:38:43.069901 (+ 89us) client_negotiation.cc:240] Negotiation successful
0618 22:38:43.070061 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":255,"spinlock_wait_cycles":25472,"thread_start_us":366,"threads_started":1}
I20260618 22:38:43.071528 7878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1793de429e384257ae03bc0e23b7da02" candidate_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e3989d8c725426da93f7200472bce7b" is_pre_election: true
I20260618 22:38:43.071527 8161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1793de429e384257ae03bc0e23b7da02" candidate_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" is_pre_election: true
I20260618 22:38:43.071981 7878 raft_consensus.cc:2454] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a7f01d4d06f4b7eba5cb53475417fd6 in term 0.
I20260618 22:38:43.071980 8161 raft_consensus.cc:2454] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a7f01d4d06f4b7eba5cb53475417fd6 in term 0.
I20260618 22:38:43.072530 7949 leader_election.cc:302] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a7f01d4d06f4b7eba5cb53475417fd6, 7e3989d8c725426da93f7200472bce7b; no voters:
I20260618 22:38:43.072873 8232 raft_consensus.cc:2781] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:38:43.072985 8232 raft_consensus.cc:482] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:38:43.073091 8232 raft_consensus.cc:3037] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Advancing to term 1
W20260618 22:38:43.074862 8068 tablet.cc:2369] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:38:43.075264 8232 raft_consensus.cc:504] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.075809 8232 leader_election.cc:288] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [CANDIDATE]: Term 1 election: Requested vote from peers 7e3989d8c725426da93f7200472bce7b (127.5.213.193:33271), eebd71fa48eb46cd8d0f506ffbff04e0 (127.5.213.195:43811)
I20260618 22:38:43.076172 7878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1793de429e384257ae03bc0e23b7da02" candidate_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e3989d8c725426da93f7200472bce7b"
I20260618 22:38:43.076251 8161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1793de429e384257ae03bc0e23b7da02" candidate_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0"
I20260618 22:38:43.076385 7878 raft_consensus.cc:3037] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:43.076457 8161 raft_consensus.cc:3037] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:43.079419 7878 raft_consensus.cc:2454] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a7f01d4d06f4b7eba5cb53475417fd6 in term 1.
I20260618 22:38:43.079419 8161 raft_consensus.cc:2454] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a7f01d4d06f4b7eba5cb53475417fd6 in term 1.
I20260618 22:38:43.079823 7951 leader_election.cc:302] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a7f01d4d06f4b7eba5cb53475417fd6, eebd71fa48eb46cd8d0f506ffbff04e0; no voters:
I20260618 22:38:43.080093 8232 raft_consensus.cc:2781] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:38:43.080474 8232 raft_consensus.cc:686] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 LEADER]: Becoming Leader. State: Replica: 0a7f01d4d06f4b7eba5cb53475417fd6, State: Running, Role: LEADER
I20260618 22:38:43.080945 8232 consensus_queue.cc:237] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.084956 7721 catalog_manager.cc:5495] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a7f01d4d06f4b7eba5cb53475417fd6 (127.5.213.194). New cstate: current_term: 1 leader_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } health_report { overall_health: UNKNOWN } } }
W20260618 22:38:43.095394 7929 tablet.cc:2369] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 18 22:38:43 dist-test-slave-lnj3 krb5kdc[7677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822321, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:43.132814 8243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.120857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33739 (local address 127.0.0.1:38826)
0618 22:38:43.121416 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:43.121431 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:43.121548 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:43.121827 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:43.121832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:43.122465 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:43.122769 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.122780 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.123710 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.123713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.124647 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.124680 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.125328 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.125332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.125444 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.126039 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:43.126056 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:43.128059 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:43.130682 (+ 2623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:43.130694 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:43.130702 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:43.131116 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:43.131424 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:43.131427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:43.131430 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:43.131563 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:43.132011 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:43.132017 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:43.132175 (+ 158us) client_negotiation.cc:769] Sending connection context
0618 22:38:43.132300 (+ 125us) client_negotiation.cc:240] Negotiation successful
0618 22:38:43.132490 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":304,"threads_started":1}
I20260618 22:38:43.132848 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.120762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.5.213.254:33739)
0618 22:38:43.120939 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:43.120944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:43.121509 (+ 565us) server_negotiation.cc:407] Connection header received
0618 22:38:43.121646 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:43.121650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:43.121707 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:43.121783 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:43.122879 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.123602 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.124768 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.125203 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.125288 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.128240 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:43.128245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:43.128248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:43.128282 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:43.130555 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:43.131201 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:43.131206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:43.131209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:43.131327 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:43.131652 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:43.131655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:43.131657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:43.131901 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:43.132004 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:43.132396 (+ 392us) server_negotiation.cc:299] Negotiation successful
0618 22:38:43.132592 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":53}
I20260618 22:38:43.137221 7721 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:38826:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260618 22:38:43.138124 7721 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:38:43.143977 7858 tablet_service.cc:1467] Processing CreateTablet for tablet 2aa9037a4775426ebda41041780b6371 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=853008af2048412290aef3554efe640a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:38:43.144402 7997 tablet_service.cc:1467] Processing CreateTablet for tablet 2aa9037a4775426ebda41041780b6371 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=853008af2048412290aef3554efe640a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:38:43.145133 7858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aa9037a4775426ebda41041780b6371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:43.145133 7997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aa9037a4775426ebda41041780b6371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:43.149556 8136 tablet_service.cc:1467] Processing CreateTablet for tablet 2aa9037a4775426ebda41041780b6371 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=853008af2048412290aef3554efe640a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:38:43.150341 8136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aa9037a4775426ebda41041780b6371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:43.152127 8225 tablet_bootstrap.cc:492] T 2aa9037a4775426ebda41041780b6371 P 0a7f01d4d06f4b7eba5cb53475417fd6: Bootstrap starting.
I20260618 22:38:43.152499 8227 tablet_bootstrap.cc:492] T 2aa9037a4775426ebda41041780b6371 P 7e3989d8c725426da93f7200472bce7b: Bootstrap starting.
I20260618 22:38:43.153716 8225 tablet_bootstrap.cc:654] T 2aa9037a4775426ebda41041780b6371 P 0a7f01d4d06f4b7eba5cb53475417fd6: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:43.153802 8227 tablet_bootstrap.cc:654] T 2aa9037a4775426ebda41041780b6371 P 7e3989d8c725426da93f7200472bce7b: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:43.155855 8227 tablet_bootstrap.cc:492] T 2aa9037a4775426ebda41041780b6371 P 7e3989d8c725426da93f7200472bce7b: No bootstrap required, opened a new log
I20260618 22:38:43.155989 8227 ts_tablet_manager.cc:1397] T 2aa9037a4775426ebda41041780b6371 P 7e3989d8c725426da93f7200472bce7b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260618 22:38:43.156422 8226 tablet_bootstrap.cc:492] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0: Bootstrap starting.
I20260618 22:38:43.156507 8225 tablet_bootstrap.cc:492] T 2aa9037a4775426ebda41041780b6371 P 0a7f01d4d06f4b7eba5cb53475417fd6: No bootstrap required, opened a new log
I20260618 22:38:43.156629 8225 ts_tablet_manager.cc:1397] T 2aa9037a4775426ebda41041780b6371 P 0a7f01d4d06f4b7eba5cb53475417fd6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260618 22:38:43.156709 8227 raft_consensus.cc:348] T 2aa9037a4775426ebda41041780b6371 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.156940 8227 raft_consensus.cc:374] T 2aa9037a4775426ebda41041780b6371 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:43.157028 8227 raft_consensus.cc:729] T 2aa9037a4775426ebda41041780b6371 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e3989d8c725426da93f7200472bce7b, State: Initialized, Role: FOLLOWER
I20260618 22:38:43.157248 8227 consensus_queue.cc:260] T 2aa9037a4775426ebda41041780b6371 P 7e3989d8c725426da93f7200472bce7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.157334 8225 raft_consensus.cc:348] T 2aa9037a4775426ebda41041780b6371 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.157534 8225 raft_consensus.cc:374] T 2aa9037a4775426ebda41041780b6371 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:43.157624 8225 raft_consensus.cc:729] T 2aa9037a4775426ebda41041780b6371 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a7f01d4d06f4b7eba5cb53475417fd6, State: Initialized, Role: FOLLOWER
I20260618 22:38:43.157812 8226 tablet_bootstrap.cc:654] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:43.157816 8227 ts_tablet_manager.cc:1428] T 2aa9037a4775426ebda41041780b6371 P 7e3989d8c725426da93f7200472bce7b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260618 22:38:43.157842 8225 consensus_queue.cc:260] T 2aa9037a4775426ebda41041780b6371 P 0a7f01d4d06f4b7eba5cb53475417fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.158197 8232 tablet_replica.cc:439] T 2aa9037a4775426ebda41041780b6371 P 0a7f01d4d06f4b7eba5cb53475417fd6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } } }
I20260618 22:38:43.158147 8233 tablet_replica.cc:439] T 2aa9037a4775426ebda41041780b6371 P 7e3989d8c725426da93f7200472bce7b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } } }
I20260618 22:38:43.158427 8232 tablet_replica.cc:442] T 2aa9037a4775426ebda41041780b6371 P 0a7f01d4d06f4b7eba5cb53475417fd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:38:43.158506 8233 tablet_replica.cc:442] T 2aa9037a4775426ebda41041780b6371 P 7e3989d8c725426da93f7200472bce7b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:38:43.159461 8225 ts_tablet_manager.cc:1428] T 2aa9037a4775426ebda41041780b6371 P 0a7f01d4d06f4b7eba5cb53475417fd6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260618 22:38:43.159789 8226 tablet_bootstrap.cc:492] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0: No bootstrap required, opened a new log
I20260618 22:38:43.159899 8226 ts_tablet_manager.cc:1397] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260618 22:38:43.160591 8226 raft_consensus.cc:348] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.160822 8226 raft_consensus.cc:374] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:43.160917 8226 raft_consensus.cc:729] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebd71fa48eb46cd8d0f506ffbff04e0, State: Initialized, Role: FOLLOWER
I20260618 22:38:43.161154 8226 consensus_queue.cc:260] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.161736 8226 ts_tablet_manager.cc:1428] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260618 22:38:43.161614 8231 tablet_replica.cc:439] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } } }
I20260618 22:38:43.161993 8231 tablet_replica.cc:442] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:38:43.418547 8231 raft_consensus.cc:482] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:38:43.418780 8231 raft_consensus.cc:504] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.420176 8231 leader_election.cc:288] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e3989d8c725426da93f7200472bce7b (127.5.213.193:33271), 0a7f01d4d06f4b7eba5cb53475417fd6 (127.5.213.194:37157)
I20260618 22:38:43.428360 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.420552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:37499 (local address 127.5.213.194:37157)
0618 22:38:43.421657 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:43.421661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:43.421676 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:38:43.421731 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:43.421734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:43.421787 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:43.421874 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:38:43.422433 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.423514 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.426081 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.427345 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.427507 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.427582 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:43.427962 (+ 380us) server_negotiation.cc:299] Negotiation successful
0618 22:38:43.428105 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1016,"thread_start_us":942,"threads_started":1}
I20260618 22:38:43.428376 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.420609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:54013 (local address 127.5.213.193:33271)
0618 22:38:43.420805 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:43.420813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:43.421222 (+ 409us) server_negotiation.cc:407] Connection header received
0618 22:38:43.421533 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:43.421539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:43.421612 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:43.421667 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:38:43.422329 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.423519 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.426065 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.427340 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.427507 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.427591 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:43.427973 (+ 382us) server_negotiation.cc:299] Negotiation successful
0618 22:38:43.428096 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260618 22:38:43.428447 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.420403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:37157 (local address 127.5.213.195:37499)
0618 22:38:43.421440 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:43.421455 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:43.421535 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:43.421913 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:43.421917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:43.421940 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:38:43.422281 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.422292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.423645 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.423650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.425906 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.425925 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.427511 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.427518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.427746 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.427818 (+ 72us) client_negotiation.cc:769] Sending connection context
0618 22:38:43.427904 (+ 86us) client_negotiation.cc:240] Negotiation successful
0618 22:38:43.428079 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":952,"mutex_wait_us":214,"thread_start_us":733,"threads_started":1}
I20260618 22:38:43.429325 8247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.420403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:33271 (local address 127.5.213.195:54013)
0618 22:38:43.421112 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:43.421131 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:43.421365 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:43.421818 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:43.421821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:43.421840 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:38:43.422083 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.422094 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.423642 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.423648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.425923 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.425932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.427548 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.427553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.427807 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.427833 (+ 26us) client_negotiation.cc:769] Sending connection context
0618 22:38:43.428609 (+ 776us) client_negotiation.cc:240] Negotiation successful
0618 22:38:43.428746 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":201,"thread_start_us":514,"threads_started":1}
I20260618 22:38:43.429370 8022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa9037a4775426ebda41041780b6371" candidate_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" is_pre_election: true
I20260618 22:38:43.429807 8022 raft_consensus.cc:2454] T 2aa9037a4775426ebda41041780b6371 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebd71fa48eb46cd8d0f506ffbff04e0 in term 0.
I20260618 22:38:43.430007 7878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa9037a4775426ebda41041780b6371" candidate_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e3989d8c725426da93f7200472bce7b" is_pre_election: true
I20260618 22:38:43.430250 7878 raft_consensus.cc:2454] T 2aa9037a4775426ebda41041780b6371 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebd71fa48eb46cd8d0f506ffbff04e0 in term 0.
I20260618 22:38:43.430377 8089 leader_election.cc:302] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a7f01d4d06f4b7eba5cb53475417fd6, eebd71fa48eb46cd8d0f506ffbff04e0; no voters:
I20260618 22:38:43.430886 8231 raft_consensus.cc:2781] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:38:43.431003 8231 raft_consensus.cc:482] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:38:43.431088 8231 raft_consensus.cc:3037] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:43.433282 8231 raft_consensus.cc:504] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.433787 8231 leader_election.cc:288] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0 [CANDIDATE]: Term 1 election: Requested vote from peers 7e3989d8c725426da93f7200472bce7b (127.5.213.193:33271), 0a7f01d4d06f4b7eba5cb53475417fd6 (127.5.213.194:37157)
I20260618 22:38:43.434173 7878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa9037a4775426ebda41041780b6371" candidate_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e3989d8c725426da93f7200472bce7b"
I20260618 22:38:43.434238 8022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa9037a4775426ebda41041780b6371" candidate_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6"
I20260618 22:38:43.434346 7878 raft_consensus.cc:3037] T 2aa9037a4775426ebda41041780b6371 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:43.434423 8022 raft_consensus.cc:3037] T 2aa9037a4775426ebda41041780b6371 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:43.436316 7878 raft_consensus.cc:2454] T 2aa9037a4775426ebda41041780b6371 P 7e3989d8c725426da93f7200472bce7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebd71fa48eb46cd8d0f506ffbff04e0 in term 1.
I20260618 22:38:43.436587 8022 raft_consensus.cc:2454] T 2aa9037a4775426ebda41041780b6371 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebd71fa48eb46cd8d0f506ffbff04e0 in term 1.
I20260618 22:38:43.436662 8088 leader_election.cc:302] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e3989d8c725426da93f7200472bce7b, eebd71fa48eb46cd8d0f506ffbff04e0; no voters:
I20260618 22:38:43.436942 8231 raft_consensus.cc:2781] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:38:43.437310 8231 raft_consensus.cc:686] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 1 LEADER]: Becoming Leader. State: Replica: eebd71fa48eb46cd8d0f506ffbff04e0, State: Running, Role: LEADER
I20260618 22:38:43.437809 8231 consensus_queue.cc:237] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.440925 8250 tablet_replica.cc:439] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0: TxnStatusTablet state changed. Reason: New leader eebd71fa48eb46cd8d0f506ffbff04e0. Latest consensus state: current_term: 1 leader_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } } }
I20260618 22:38:43.441277 8250 tablet_replica.cc:442] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:38:43.441862 8252 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260618 22:38:43.442090 8252 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260618 22:38:43.442277 7721 catalog_manager.cc:5495] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to eebd71fa48eb46cd8d0f506ffbff04e0 (127.5.213.195). New cstate: current_term: 1 leader_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } health_report { overall_health: HEALTHY } } }
I20260618 22:38:43.464382 8232 consensus_queue.cc:1035] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:38:43.472119 8232 consensus_queue.cc:1035] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:38:43.528651 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.522677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.5.213.195:43811)
0618 22:38:43.522863 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:43.522869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:43.523152 (+ 283us) server_negotiation.cc:407] Connection header received
0618 22:38:43.523238 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:43.523242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:43.523320 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:43.523423 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:38:43.523817 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.524712 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.526565 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.526996 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.527101 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.527571 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:38:43.528101 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:38:43.528202 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:43.528340 (+ 138us) server_negotiation.cc:299] Negotiation successful
0618 22:38:43.528420 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260618 22:38:43.528759 8243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.522618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:43811 (local address 127.0.0.1:59584)
0618 22:38:43.522885 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:43.522904 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:43.523028 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:43.523472 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:43.523477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:43.523490 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:38:43.523674 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.523685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.524830 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.524836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.526418 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.526433 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.527140 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.527146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.527333 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.527412 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:38:43.528193 (+ 781us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:38:43.528215 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:38:43.528304 (+ 89us) client_negotiation.cc:240] Negotiation successful
0618 22:38:43.528462 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260618 22:38:43.536114 7878 raft_consensus.cc:1261] T 2aa9037a4775426ebda41041780b6371 P 7e3989d8c725426da93f7200472bce7b [term 1 FOLLOWER]: Refusing update from remote peer eebd71fa48eb46cd8d0f506ffbff04e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:38:43.536391 8022 raft_consensus.cc:1261] T 2aa9037a4775426ebda41041780b6371 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 FOLLOWER]: Refusing update from remote peer eebd71fa48eb46cd8d0f506ffbff04e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:38:43.536736 8250 consensus_queue.cc:1035] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:38:43.537086 8231 consensus_queue.cc:1035] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:38:43.543819 8232 tablet_replica.cc:439] T 2aa9037a4775426ebda41041780b6371 P 0a7f01d4d06f4b7eba5cb53475417fd6: TxnStatusTablet state changed. Reason: New leader eebd71fa48eb46cd8d0f506ffbff04e0. Latest consensus state: current_term: 1 leader_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } } }
I20260618 22:38:43.544131 8232 tablet_replica.cc:442] T 2aa9037a4775426ebda41041780b6371 P 0a7f01d4d06f4b7eba5cb53475417fd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:38:43.546584 8258 mvcc.cc:205] Tried to move back new op lower bound from 7298344237195296768 to 7298344236804775936. Current Snapshot: MvccSnapshot[applied={T|T < 7298344237195296768}]
I20260618 22:38:43.547791 8231 tablet_replica.cc:439] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } } }
I20260618 22:38:43.548058 8231 tablet_replica.cc:442] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:38:43.548491 8232 tablet_replica.cc:439] T 2aa9037a4775426ebda41041780b6371 P 0a7f01d4d06f4b7eba5cb53475417fd6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } } }
I20260618 22:38:43.548754 8232 tablet_replica.cc:442] T 2aa9037a4775426ebda41041780b6371 P 0a7f01d4d06f4b7eba5cb53475417fd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:38:43.552718 8233 tablet_replica.cc:439] T 2aa9037a4775426ebda41041780b6371 P 7e3989d8c725426da93f7200472bce7b: TxnStatusTablet state changed. Reason: New leader eebd71fa48eb46cd8d0f506ffbff04e0. Latest consensus state: current_term: 1 leader_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } } }
I20260618 22:38:43.552982 8233 tablet_replica.cc:442] T 2aa9037a4775426ebda41041780b6371 P 7e3989d8c725426da93f7200472bce7b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:38:43.554805 8250 tablet_replica.cc:439] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } } }
I20260618 22:38:43.555068 8250 tablet_replica.cc:442] T 2aa9037a4775426ebda41041780b6371 P eebd71fa48eb46cd8d0f506ffbff04e0: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:38:43.562311 8233 tablet_replica.cc:439] T 2aa9037a4775426ebda41041780b6371 P 7e3989d8c725426da93f7200472bce7b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } } }
I20260618 22:38:43.562551 8233 tablet_replica.cc:442] T 2aa9037a4775426ebda41041780b6371 P 7e3989d8c725426da93f7200472bce7b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:38:43.567763 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.559925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54698 (local address 127.5.213.194:37157)
0618 22:38:43.560149 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:43.560155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:43.560349 (+ 194us) server_negotiation.cc:407] Connection header received
0618 22:38:43.560657 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:43.560663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:43.560775 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:43.560919 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:38:43.561582 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.562705 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.564774 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.565534 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.565634 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.566301 (+ 667us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:38:43.567051 (+ 750us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:38:43.567168 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:43.567449 (+ 281us) server_negotiation.cc:299] Negotiation successful
0618 22:38:43.567526 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260618 22:38:43.583315 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.577383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:43811 (local address 127.5.213.194:42957)
0618 22:38:43.577949 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:43.577967 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:43.578079 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:43.578379 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:43.578382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:43.578390 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:38:43.578587 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.578594 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.580019 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.580026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.581215 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.581230 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.582230 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.582235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.582357 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.582412 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:38:43.582848 (+ 436us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:38:43.582867 (+ 19us) client_negotiation.cc:769] Sending connection context
0618 22:38:43.582945 (+ 78us) client_negotiation.cc:240] Negotiation successful
0618 22:38:43.583068 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":283,"threads_started":1}
I20260618 22:38:43.583369 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.577526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:42957 (local address 127.5.213.195:43811)
0618 22:38:43.577732 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:43.577738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:43.578041 (+ 303us) server_negotiation.cc:407] Connection header received
0618 22:38:43.578187 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:43.578193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:43.578276 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:43.578378 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:38:43.578761 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.579868 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.581364 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.582062 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.582197 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.582596 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:38:43.582755 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:38:43.582839 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:43.582978 (+ 139us) server_negotiation.cc:299] Negotiation successful
0618 22:38:43.583060 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260618 22:38:43.641204 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.622729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:37219 (local address 127.5.213.194:37157)
0618 22:38:43.622913 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:43.622919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:43.635528 (+ 12609us) server_negotiation.cc:407] Connection header received
0618 22:38:43.635715 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:43.635720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:43.635830 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:43.635916 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:38:43.636692 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.637647 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.639178 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.639636 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.639749 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.640250 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:38:43.640366 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:38:43.640439 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:43.640846 (+ 407us) server_negotiation.cc:299] Negotiation successful
0618 22:38:43.640922 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:38:43.641346 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.622610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:37157 (local address 127.5.213.195:37219)
0618 22:38:43.635403 (+ 12793us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:43.635421 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:43.635574 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:43.636079 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:43.636083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:43.636095 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:38:43.636540 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.636553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.637782 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.637788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.639049 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.639060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.639827 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.639834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.640014 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.640080 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:38:43.640717 (+ 637us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:38:43.640726 (+ 9us) client_negotiation.cc:769] Sending connection context
0618 22:38:43.640805 (+ 79us) client_negotiation.cc:240] Negotiation successful
0618 22:38:43.640935 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12565,"thread_start_us":13031,"threads_started":1}
I20260618 22:38:43.683596 7722 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:38:43.683902 7722 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:38:43.687750 7722 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d: Sending DeleteTablet for 3 replicas of tablet 1793de429e384257ae03bc0e23b7da02
I20260618 22:38:43.688868 7858 tablet_service.cc:1514] Processing DeleteTablet for tablet 1793de429e384257ae03bc0e23b7da02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:38:43 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:46294
I20260618 22:38:43.689688 8277 tablet_replica.cc:330] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b: stopping tablet replica
I20260618 22:38:43.690143 7997 tablet_service.cc:1514] Processing DeleteTablet for tablet 1793de429e384257ae03bc0e23b7da02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:38:43 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:54686
I20260618 22:38:43.690203 8277 raft_consensus.cc:2229] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:38:43.690622 8277 raft_consensus.cc:2258] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:38:43.690878 8136 tablet_service.cc:1514] Processing DeleteTablet for tablet 1793de429e384257ae03bc0e23b7da02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:38:43 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:59582
I20260618 22:38:43.691660 8279 tablet_replica.cc:330] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0: stopping tablet replica
I20260618 22:38:43.692101 8279 raft_consensus.cc:2229] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:38:43.692457 8279 raft_consensus.cc:2258] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:38:43.692602 8277 ts_tablet_manager.cc:1905] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:38:43.692945 7722 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:38:43.693567 7722 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:38:43.699653 8277 ts_tablet_manager.cc:1918] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:38:43.699815 8277 log.cc:1196] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/1793de429e384257ae03bc0e23b7da02
I20260618 22:38:43.701419 7858 tablet_service.cc:1467] Processing CreateTablet for tablet d8e992767626442a9243f1fe894f3d29 (DEFAULT_TABLE table=test-table [id=ff63076be05d49abbea16b032f05210a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:43.701831 7858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e992767626442a9243f1fe894f3d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:43.705866 8277 ts_tablet_manager.cc:1939] T 1793de429e384257ae03bc0e23b7da02 P 7e3989d8c725426da93f7200472bce7b: Deleting consensus metadata
I20260618 22:38:43.707397 7996 tablet_service.cc:1467] Processing CreateTablet for tablet d8e992767626442a9243f1fe894f3d29 (DEFAULT_TABLE table=test-table [id=ff63076be05d49abbea16b032f05210a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:43.707944 7996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e992767626442a9243f1fe894f3d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:43.708082 7707 catalog_manager.cc:4841] TS 7e3989d8c725426da93f7200472bce7b (127.5.213.193:33271): tablet 1793de429e384257ae03bc0e23b7da02 (table test-table [id=da14333fed4449159e08296adeef01c3]) successfully deleted
I20260618 22:38:43.708777 8136 tablet_service.cc:1467] Processing CreateTablet for tablet d8e992767626442a9243f1fe894f3d29 (DEFAULT_TABLE table=test-table [id=ff63076be05d49abbea16b032f05210a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:43.710538 8136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e992767626442a9243f1fe894f3d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:43.717576 8281 tablet_bootstrap.cc:492] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b: Bootstrap starting.
I20260618 22:38:43.719321 8281 tablet_bootstrap.cc:654] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:43.725308 8283 tablet_bootstrap.cc:492] T d8e992767626442a9243f1fe894f3d29 P eebd71fa48eb46cd8d0f506ffbff04e0: Bootstrap starting.
I20260618 22:38:43.725998 8279 ts_tablet_manager.cc:1905] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:38:43.726583 8281 tablet_bootstrap.cc:492] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b: No bootstrap required, opened a new log
I20260618 22:38:43.726755 8281 ts_tablet_manager.cc:1397] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260618 22:38:43.727509 8281 raft_consensus.cc:348] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.727766 8281 raft_consensus.cc:374] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:43.727860 8281 raft_consensus.cc:729] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e3989d8c725426da93f7200472bce7b, State: Initialized, Role: FOLLOWER
I20260618 22:38:43.731719 8278 tablet_replica.cc:330] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6: stopping tablet replica
I20260618 22:38:43.731932 8281 consensus_queue.cc:260] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.732695 8278 raft_consensus.cc:2229] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:38:43.732923 8283 tablet_bootstrap.cc:654] T d8e992767626442a9243f1fe894f3d29 P eebd71fa48eb46cd8d0f506ffbff04e0: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:43.733528 8281 ts_tablet_manager.cc:1428] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260618 22:38:43.731791 8280 tablet_bootstrap.cc:492] T d8e992767626442a9243f1fe894f3d29 P 0a7f01d4d06f4b7eba5cb53475417fd6: Bootstrap starting.
I20260618 22:38:43.735095 8278 raft_consensus.cc:2258] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:38:43.736941 8280 tablet_bootstrap.cc:654] T d8e992767626442a9243f1fe894f3d29 P 0a7f01d4d06f4b7eba5cb53475417fd6: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:43.739029 8278 ts_tablet_manager.cc:1905] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:38:43.740289 8280 tablet_bootstrap.cc:492] T d8e992767626442a9243f1fe894f3d29 P 0a7f01d4d06f4b7eba5cb53475417fd6: No bootstrap required, opened a new log
I20260618 22:38:43.740439 8280 ts_tablet_manager.cc:1397] T d8e992767626442a9243f1fe894f3d29 P 0a7f01d4d06f4b7eba5cb53475417fd6: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260618 22:38:43.741250 8280 raft_consensus.cc:348] T d8e992767626442a9243f1fe894f3d29 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.741506 8280 raft_consensus.cc:374] T d8e992767626442a9243f1fe894f3d29 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:43.741613 8280 raft_consensus.cc:729] T d8e992767626442a9243f1fe894f3d29 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a7f01d4d06f4b7eba5cb53475417fd6, State: Initialized, Role: FOLLOWER
I20260618 22:38:43.741884 8280 consensus_queue.cc:260] T d8e992767626442a9243f1fe894f3d29 P 0a7f01d4d06f4b7eba5cb53475417fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.742659 8280 ts_tablet_manager.cc:1428] T d8e992767626442a9243f1fe894f3d29 P 0a7f01d4d06f4b7eba5cb53475417fd6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260618 22:38:43.747877 8278 ts_tablet_manager.cc:1918] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:38:43.748036 8278 log.cc:1196] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/1793de429e384257ae03bc0e23b7da02
I20260618 22:38:43.748457 8278 ts_tablet_manager.cc:1939] T 1793de429e384257ae03bc0e23b7da02 P 0a7f01d4d06f4b7eba5cb53475417fd6: Deleting consensus metadata
I20260618 22:38:43.749758 7708 catalog_manager.cc:4841] TS 0a7f01d4d06f4b7eba5cb53475417fd6 (127.5.213.194:37157): tablet 1793de429e384257ae03bc0e23b7da02 (table test-table [id=da14333fed4449159e08296adeef01c3]) successfully deleted
I20260618 22:38:43.751688 8283 tablet_bootstrap.cc:492] T d8e992767626442a9243f1fe894f3d29 P eebd71fa48eb46cd8d0f506ffbff04e0: No bootstrap required, opened a new log
I20260618 22:38:43.751832 8283 ts_tablet_manager.cc:1397] T d8e992767626442a9243f1fe894f3d29 P eebd71fa48eb46cd8d0f506ffbff04e0: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20260618 22:38:43.752485 8283 raft_consensus.cc:348] T d8e992767626442a9243f1fe894f3d29 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.752684 8283 raft_consensus.cc:374] T d8e992767626442a9243f1fe894f3d29 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:43.752775 8283 raft_consensus.cc:729] T d8e992767626442a9243f1fe894f3d29 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebd71fa48eb46cd8d0f506ffbff04e0, State: Initialized, Role: FOLLOWER
I20260618 22:38:43.752857 8279 ts_tablet_manager.cc:1918] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:38:43.753032 8279 log.cc:1196] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/1793de429e384257ae03bc0e23b7da02
I20260618 22:38:43.753010 8283 consensus_queue.cc:260] T d8e992767626442a9243f1fe894f3d29 P eebd71fa48eb46cd8d0f506ffbff04e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.753571 8283 ts_tablet_manager.cc:1428] T d8e992767626442a9243f1fe894f3d29 P eebd71fa48eb46cd8d0f506ffbff04e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260618 22:38:43.753598 8279 ts_tablet_manager.cc:1939] T 1793de429e384257ae03bc0e23b7da02 P eebd71fa48eb46cd8d0f506ffbff04e0: Deleting consensus metadata
I20260618 22:38:43.755424 7709 catalog_manager.cc:4841] TS eebd71fa48eb46cd8d0f506ffbff04e0 (127.5.213.195:43811): tablet 1793de429e384257ae03bc0e23b7da02 (table test-table [id=da14333fed4449159e08296adeef01c3]) successfully deleted
I20260618 22:38:43.821583 8233 raft_consensus.cc:482] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:38:43.821828 8233 raft_consensus.cc:504] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.823473 8233 leader_election.cc:288] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a7f01d4d06f4b7eba5cb53475417fd6 (127.5.213.194:37157), eebd71fa48eb46cd8d0f506ffbff04e0 (127.5.213.195:43811)
I20260618 22:38:43.848568 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.824844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:38457 (local address 127.5.213.195:43811)
0618 22:38:43.825042 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:43.825048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:43.841340 (+ 16292us) server_negotiation.cc:407] Connection header received
0618 22:38:43.841588 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:43.841594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:43.841673 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:43.841727 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:38:43.842462 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.843590 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.846030 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.847405 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.847525 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.847599 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:43.848119 (+ 520us) server_negotiation.cc:299] Negotiation successful
0618 22:38:43.848269 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 22:38:43.849354 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.824377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:43811 (local address 127.5.213.193:38457)
0618 22:38:43.841244 (+ 16867us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:43.841263 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:43.841434 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:43.841889 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:43.841894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:43.841917 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:38:43.842282 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.842291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.843766 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.843771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.845841 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.845854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.847757 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.847762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.847950 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.847982 (+ 32us) client_negotiation.cc:769] Sending connection context
0618 22:38:43.848836 (+ 854us) client_negotiation.cc:240] Negotiation successful
0618 22:38:43.848974 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16496,"thread_start_us":11371,"threads_started":1}
I20260618 22:38:43.850006 8161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e992767626442a9243f1fe894f3d29" candidate_uuid: "7e3989d8c725426da93f7200472bce7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" is_pre_election: true
I20260618 22:38:43.850281 8161 raft_consensus.cc:2454] T d8e992767626442a9243f1fe894f3d29 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e3989d8c725426da93f7200472bce7b in term 0.
I20260618 22:38:43.850812 7812 leader_election.cc:302] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e3989d8c725426da93f7200472bce7b, eebd71fa48eb46cd8d0f506ffbff04e0; no voters:
I20260618 22:38:43.851189 8233 raft_consensus.cc:2781] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:38:43.851305 8233 raft_consensus.cc:482] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:38:43.851393 8233 raft_consensus.cc:3037] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:43.853358 8233 raft_consensus.cc:504] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.853945 8233 leader_election.cc:288] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [CANDIDATE]: Term 1 election: Requested vote from peers 0a7f01d4d06f4b7eba5cb53475417fd6 (127.5.213.194:37157), eebd71fa48eb46cd8d0f506ffbff04e0 (127.5.213.195:43811)
I20260618 22:38:43.856328 8161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e992767626442a9243f1fe894f3d29" candidate_uuid: "7e3989d8c725426da93f7200472bce7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0"
I20260618 22:38:43.856560 8161 raft_consensus.cc:3037] T d8e992767626442a9243f1fe894f3d29 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:43.859614 8161 raft_consensus.cc:2454] T d8e992767626442a9243f1fe894f3d29 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e3989d8c725426da93f7200472bce7b in term 1.
I20260618 22:38:43.860093 7812 leader_election.cc:302] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e3989d8c725426da93f7200472bce7b, eebd71fa48eb46cd8d0f506ffbff04e0; no voters:
I20260618 22:38:43.860483 8233 raft_consensus.cc:2781] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:38:43.864140 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.823865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:54453 (local address 127.5.213.194:37157)
0618 22:38:43.824054 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:43.824060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:43.835733 (+ 11673us) server_negotiation.cc:407] Connection header received
0618 22:38:43.836236 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:43.836242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:43.836346 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:43.836410 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:38:43.837319 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.838419 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.854164 (+ 15745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.855527 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.855696 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.855771 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:43.863676 (+ 7905us) server_negotiation.cc:299] Negotiation successful
0618 22:38:43.863837 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:38:43.864905 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.823648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:37157 (local address 127.5.213.193:54453)
0618 22:38:43.835585 (+ 11937us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:43.835608 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:43.836055 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:43.836609 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:43.836614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:43.836641 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:38:43.837131 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.837145 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.838608 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.838613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.853753 (+ 15140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.853767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.862776 (+ 9009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.862783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.862980 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.863016 (+ 36us) client_negotiation.cc:769] Sending connection context
0618 22:38:43.864386 (+ 1370us) client_negotiation.cc:240] Negotiation successful
0618 22:38:43.864539 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11774,"thread_start_us":17412,"threads_started":1}
I20260618 22:38:43.865566 8233 raft_consensus.cc:686] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [term 1 LEADER]: Becoming Leader. State: Replica: 7e3989d8c725426da93f7200472bce7b, State: Running, Role: LEADER
I20260618 22:38:43.865535 8021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e992767626442a9243f1fe894f3d29" candidate_uuid: "7e3989d8c725426da93f7200472bce7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" is_pre_election: true
I20260618 22:38:43.865809 8021 raft_consensus.cc:2454] T d8e992767626442a9243f1fe894f3d29 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e3989d8c725426da93f7200472bce7b in term 0.
I20260618 22:38:43.866143 8022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e992767626442a9243f1fe894f3d29" candidate_uuid: "7e3989d8c725426da93f7200472bce7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6"
I20260618 22:38:43.866323 8022 raft_consensus.cc:3037] T d8e992767626442a9243f1fe894f3d29 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:43.866222 8233 consensus_queue.cc:237] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:43.869400 8022 raft_consensus.cc:2454] T d8e992767626442a9243f1fe894f3d29 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e3989d8c725426da93f7200472bce7b in term 1.
I20260618 22:38:43.875921 7722 catalog_manager.cc:5495] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e3989d8c725426da93f7200472bce7b (127.5.213.193). New cstate: current_term: 1 leader_uuid: "7e3989d8c725426da93f7200472bce7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } health_report { overall_health: UNKNOWN } } }
I20260618 22:38:43.924515 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.918918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46306 (local address 127.5.213.193:33271)
0618 22:38:43.919071 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:43.919077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:43.919273 (+ 196us) server_negotiation.cc:407] Connection header received
0618 22:38:43.919468 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:43.919472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:43.919555 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:43.919690 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:38:43.920022 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.920757 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.922315 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.922858 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.922964 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.923378 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:38:43.923892 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:38:43.924020 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:43.924104 (+ 84us) server_negotiation.cc:299] Negotiation successful
0618 22:38:43.924190 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:38:43.940376 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.934547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:46323 (local address 127.5.213.195:43811)
0618 22:38:43.934761 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:43.934768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:43.935331 (+ 563us) server_negotiation.cc:407] Connection header received
0618 22:38:43.935503 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:43.935507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:43.935568 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:43.935637 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:38:43.936192 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.936879 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.938778 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.939166 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.939290 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.939812 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:38:43.939920 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:38:43.939981 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:43.940114 (+ 133us) server_negotiation.cc:299] Negotiation successful
0618 22:38:43.940174 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:38:43.940605 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.934514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:43811 (local address 127.5.213.193:46323)
0618 22:38:43.935236 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:43.935256 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:43.935398 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:43.935668 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:43.935673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:43.935685 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:38:43.936045 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.936056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.937003 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.937008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.938655 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.938670 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.939349 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.939355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.939549 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.939616 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:38:43.940005 (+ 389us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:38:43.940017 (+ 12us) client_negotiation.cc:769] Sending connection context
0618 22:38:43.940100 (+ 83us) client_negotiation.cc:240] Negotiation successful
0618 22:38:43.940235 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":369,"threads_started":1}
I20260618 22:38:43.948369 8161 raft_consensus.cc:1261] T d8e992767626442a9243f1fe894f3d29 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 1 FOLLOWER]: Refusing update from remote peer 7e3989d8c725426da93f7200472bce7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:38:43.948513 8022 raft_consensus.cc:1261] T d8e992767626442a9243f1fe894f3d29 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 FOLLOWER]: Refusing update from remote peer 7e3989d8c725426da93f7200472bce7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:38:43.949038 8288 consensus_queue.cc:1035] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:38:43.949352 8233 consensus_queue.cc:1035] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:38:43.953042 8256 mvcc.cc:205] Tried to move back new op lower bound from 7298344238887378944 to 7298344238562308096. Current Snapshot: MvccSnapshot[applied={T|T < 7298344238887378944}]
I20260618 22:38:43.976908 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.971126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:33271 (local address 127.5.213.195:36117)
0618 22:38:43.971340 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:43.971358 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:43.971460 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:43.971786 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:43.971790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:43.971802 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:38:43.971969 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.971981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.973249 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.973254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.974749 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:43.974762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.975754 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.975760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:43.975929 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.975972 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:38:43.976393 (+ 421us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:38:43.976403 (+ 10us) client_negotiation.cc:769] Sending connection context
0618 22:38:43.976469 (+ 66us) client_negotiation.cc:240] Negotiation successful
0618 22:38:43.976597 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260618 22:38:43.977558 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:43.971131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:36117 (local address 127.5.213.193:33271)
0618 22:38:43.971526 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:43.971533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:43.971550 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:38:43.971605 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:43.971609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:43.971683 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:43.971769 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:38:43.972118 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.973101 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.974905 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:43.975518 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:43.975632 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:43.976131 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:38:43.976256 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:38:43.976346 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:43.977208 (+ 862us) server_negotiation.cc:299] Negotiation successful
0618 22:38:43.977280 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274}
W20260618 22:38:43.978430 8256 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260618 22:38:45.575487 7722 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:38:45.575724 7722 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:38:45.579607 7722 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d: Sending DeleteTablet for 3 replicas of tablet d8e992767626442a9243f1fe894f3d29
I20260618 22:38:45.580374 7858 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e992767626442a9243f1fe894f3d29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:38:45 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:46294
I20260618 22:38:45.580439 7997 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e992767626442a9243f1fe894f3d29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:38:45 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:54686
I20260618 22:38:45.580627 8136 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e992767626442a9243f1fe894f3d29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:38:45 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:59582
I20260618 22:38:45.581105 8320 tablet_replica.cc:330] T d8e992767626442a9243f1fe894f3d29 P 0a7f01d4d06f4b7eba5cb53475417fd6: stopping tablet replica
I20260618 22:38:45.581344 8320 raft_consensus.cc:2229] T d8e992767626442a9243f1fe894f3d29 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:38:45.581961 8320 raft_consensus.cc:2258] T d8e992767626442a9243f1fe894f3d29 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:38:45.587591 8320 ts_tablet_manager.cc:1905] T d8e992767626442a9243f1fe894f3d29 P 0a7f01d4d06f4b7eba5cb53475417fd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:38:45.593128 8319 tablet_replica.cc:330] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b: stopping tablet replica
I20260618 22:38:45.593339 8319 raft_consensus.cc:2229] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:38:45.593854 8319 raft_consensus.cc:2258] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:38:45.594403 8321 tablet_replica.cc:330] T d8e992767626442a9243f1fe894f3d29 P eebd71fa48eb46cd8d0f506ffbff04e0: stopping tablet replica
I20260618 22:38:45.594760 8320 ts_tablet_manager.cc:1918] T d8e992767626442a9243f1fe894f3d29 P 0a7f01d4d06f4b7eba5cb53475417fd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 22:38:45.594899 8320 log.cc:1196] T d8e992767626442a9243f1fe894f3d29 P 0a7f01d4d06f4b7eba5cb53475417fd6: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/d8e992767626442a9243f1fe894f3d29
I20260618 22:38:45.594885 8321 raft_consensus.cc:2229] T d8e992767626442a9243f1fe894f3d29 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:38:45.595116 8321 raft_consensus.cc:2258] T d8e992767626442a9243f1fe894f3d29 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:38:45.595291 8320 ts_tablet_manager.cc:1939] T d8e992767626442a9243f1fe894f3d29 P 0a7f01d4d06f4b7eba5cb53475417fd6: Deleting consensus metadata
I20260618 22:38:45.595669 8319 ts_tablet_manager.cc:1905] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:38:45.596033 8321 ts_tablet_manager.cc:1905] T d8e992767626442a9243f1fe894f3d29 P eebd71fa48eb46cd8d0f506ffbff04e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:38:45.596423 7708 catalog_manager.cc:4841] TS 0a7f01d4d06f4b7eba5cb53475417fd6 (127.5.213.194:37157): tablet d8e992767626442a9243f1fe894f3d29 (table test-table [id=ff63076be05d49abbea16b032f05210a]) successfully deleted
I20260618 22:38:45.600684 8319 ts_tablet_manager.cc:1918] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 22:38:45.600798 8319 log.cc:1196] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/d8e992767626442a9243f1fe894f3d29
I20260618 22:38:45.601231 8319 ts_tablet_manager.cc:1939] T d8e992767626442a9243f1fe894f3d29 P 7e3989d8c725426da93f7200472bce7b: Deleting consensus metadata
I20260618 22:38:45.602089 8321 ts_tablet_manager.cc:1918] T d8e992767626442a9243f1fe894f3d29 P eebd71fa48eb46cd8d0f506ffbff04e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 22:38:45.602238 8321 log.cc:1196] T d8e992767626442a9243f1fe894f3d29 P eebd71fa48eb46cd8d0f506ffbff04e0: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/d8e992767626442a9243f1fe894f3d29
I20260618 22:38:45.602420 7707 catalog_manager.cc:4841] TS 7e3989d8c725426da93f7200472bce7b (127.5.213.193:33271): tablet d8e992767626442a9243f1fe894f3d29 (table test-table [id=ff63076be05d49abbea16b032f05210a]) successfully deleted
I20260618 22:38:45.602619 8321 ts_tablet_manager.cc:1939] T d8e992767626442a9243f1fe894f3d29 P eebd71fa48eb46cd8d0f506ffbff04e0: Deleting consensus metadata
I20260618 22:38:45.603488 7709 catalog_manager.cc:4841] TS eebd71fa48eb46cd8d0f506ffbff04e0 (127.5.213.195:43811): tablet d8e992767626442a9243f1fe894f3d29 (table test-table [id=ff63076be05d49abbea16b032f05210a]) successfully deleted
Jun 18 22:38:45 dist-test-slave-lnj3 krb5kdc[7677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822325, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 22:38:45 dist-test-slave-lnj3 krb5kdc[7677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822325, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:45.632565 8329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:45.619533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38842 (local address 127.5.213.254:33739)
0618 22:38:45.620032 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:45.620036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:45.620059 (+ 23us) server_negotiation.cc:407] Connection header received
0618 22:38:45.620283 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:45.620287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:45.620381 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:45.620481 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:45.621827 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:45.622708 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:45.623896 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:45.624283 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:45.624376 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:45.627807 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:45.627812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:45.627815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:45.627848 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:45.630406 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:45.631002 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:45.631006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:45.631008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:45.631120 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:45.631402 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:45.631405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:45.631406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:45.631656 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:45.631815 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:45.632106 (+ 291us) server_negotiation.cc:299] Negotiation successful
0618 22:38:45.632319 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":271,"threads_started":1}
I20260618 22:38:45.635931 7722 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:38:45.636487 7722 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:38:45.642311 7858 tablet_service.cc:1467] Processing CreateTablet for tablet a881dad2cce440f984b5af0d3b3ec6dd (DEFAULT_TABLE table=test-table [id=7a55a8c5ee954a5ea0386e49b9005acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:45.642371 7997 tablet_service.cc:1467] Processing CreateTablet for tablet a881dad2cce440f984b5af0d3b3ec6dd (DEFAULT_TABLE table=test-table [id=7a55a8c5ee954a5ea0386e49b9005acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:45.642563 8136 tablet_service.cc:1467] Processing CreateTablet for tablet a881dad2cce440f984b5af0d3b3ec6dd (DEFAULT_TABLE table=test-table [id=7a55a8c5ee954a5ea0386e49b9005acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:45.642782 7858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a881dad2cce440f984b5af0d3b3ec6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:45.642791 7997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a881dad2cce440f984b5af0d3b3ec6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:45.642942 8136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a881dad2cce440f984b5af0d3b3ec6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:45.647830 8330 tablet_bootstrap.cc:492] T a881dad2cce440f984b5af0d3b3ec6dd P 7e3989d8c725426da93f7200472bce7b: Bootstrap starting.
I20260618 22:38:45.649088 8331 tablet_bootstrap.cc:492] T a881dad2cce440f984b5af0d3b3ec6dd P eebd71fa48eb46cd8d0f506ffbff04e0: Bootstrap starting.
I20260618 22:38:45.649428 8330 tablet_bootstrap.cc:654] T a881dad2cce440f984b5af0d3b3ec6dd P 7e3989d8c725426da93f7200472bce7b: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:45.650660 8331 tablet_bootstrap.cc:654] T a881dad2cce440f984b5af0d3b3ec6dd P eebd71fa48eb46cd8d0f506ffbff04e0: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:45.650918 8332 tablet_bootstrap.cc:492] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6: Bootstrap starting.
I20260618 22:38:45.652396 8332 tablet_bootstrap.cc:654] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:45.652768 8330 tablet_bootstrap.cc:492] T a881dad2cce440f984b5af0d3b3ec6dd P 7e3989d8c725426da93f7200472bce7b: No bootstrap required, opened a new log
I20260618 22:38:45.652909 8330 ts_tablet_manager.cc:1397] T a881dad2cce440f984b5af0d3b3ec6dd P 7e3989d8c725426da93f7200472bce7b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260618 22:38:45.652954 8331 tablet_bootstrap.cc:492] T a881dad2cce440f984b5af0d3b3ec6dd P eebd71fa48eb46cd8d0f506ffbff04e0: No bootstrap required, opened a new log
I20260618 22:38:45.653079 8331 ts_tablet_manager.cc:1397] T a881dad2cce440f984b5af0d3b3ec6dd P eebd71fa48eb46cd8d0f506ffbff04e0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260618 22:38:45.653589 8330 raft_consensus.cc:348] T a881dad2cce440f984b5af0d3b3ec6dd P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:45.653781 8330 raft_consensus.cc:374] T a881dad2cce440f984b5af0d3b3ec6dd P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:45.653870 8330 raft_consensus.cc:729] T a881dad2cce440f984b5af0d3b3ec6dd P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e3989d8c725426da93f7200472bce7b, State: Initialized, Role: FOLLOWER
I20260618 22:38:45.653791 8331 raft_consensus.cc:348] T a881dad2cce440f984b5af0d3b3ec6dd P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:45.653995 8331 raft_consensus.cc:374] T a881dad2cce440f984b5af0d3b3ec6dd P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:45.654083 8331 raft_consensus.cc:729] T a881dad2cce440f984b5af0d3b3ec6dd P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebd71fa48eb46cd8d0f506ffbff04e0, State: Initialized, Role: FOLLOWER
I20260618 22:38:45.654120 8330 consensus_queue.cc:260] T a881dad2cce440f984b5af0d3b3ec6dd P 7e3989d8c725426da93f7200472bce7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:45.654389 8332 tablet_bootstrap.cc:492] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6: No bootstrap required, opened a new log
I20260618 22:38:45.654505 8332 ts_tablet_manager.cc:1397] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260618 22:38:45.654294 8331 consensus_queue.cc:260] T a881dad2cce440f984b5af0d3b3ec6dd P eebd71fa48eb46cd8d0f506ffbff04e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:45.654657 8330 ts_tablet_manager.cc:1428] T a881dad2cce440f984b5af0d3b3ec6dd P 7e3989d8c725426da93f7200472bce7b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260618 22:38:45.655407 8331 ts_tablet_manager.cc:1428] T a881dad2cce440f984b5af0d3b3ec6dd P eebd71fa48eb46cd8d0f506ffbff04e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260618 22:38:45.655390 8332 raft_consensus.cc:348] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:45.655602 8332 raft_consensus.cc:374] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:45.655689 8332 raft_consensus.cc:729] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a7f01d4d06f4b7eba5cb53475417fd6, State: Initialized, Role: FOLLOWER
I20260618 22:38:45.655944 8332 consensus_queue.cc:260] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:45.656658 8332 ts_tablet_manager.cc:1428] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260618 22:38:45.772588 8336 raft_consensus.cc:482] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:38:45.772809 8336 raft_consensus.cc:504] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:45.773469 8336 leader_election.cc:288] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e3989d8c725426da93f7200472bce7b (127.5.213.193:33271), eebd71fa48eb46cd8d0f506ffbff04e0 (127.5.213.195:43811)
I20260618 22:38:45.774050 7878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a881dad2cce440f984b5af0d3b3ec6dd" candidate_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e3989d8c725426da93f7200472bce7b" is_pre_election: true
I20260618 22:38:45.774055 8160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a881dad2cce440f984b5af0d3b3ec6dd" candidate_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" is_pre_election: true
I20260618 22:38:45.774313 8160 raft_consensus.cc:2454] T a881dad2cce440f984b5af0d3b3ec6dd P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a7f01d4d06f4b7eba5cb53475417fd6 in term 0.
I20260618 22:38:45.774338 7878 raft_consensus.cc:2454] T a881dad2cce440f984b5af0d3b3ec6dd P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a7f01d4d06f4b7eba5cb53475417fd6 in term 0.
I20260618 22:38:45.774791 7949 leader_election.cc:302] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0a7f01d4d06f4b7eba5cb53475417fd6, 7e3989d8c725426da93f7200472bce7b, eebd71fa48eb46cd8d0f506ffbff04e0; no voters:
I20260618 22:38:45.775056 8336 raft_consensus.cc:2781] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:38:45.775162 8336 raft_consensus.cc:482] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:38:45.775249 8336 raft_consensus.cc:3037] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:45.777202 8336 raft_consensus.cc:504] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:45.777663 8336 leader_election.cc:288] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [CANDIDATE]: Term 1 election: Requested vote from peers 7e3989d8c725426da93f7200472bce7b (127.5.213.193:33271), eebd71fa48eb46cd8d0f506ffbff04e0 (127.5.213.195:43811)
I20260618 22:38:45.778115 8160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a881dad2cce440f984b5af0d3b3ec6dd" candidate_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0"
I20260618 22:38:45.778103 7878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a881dad2cce440f984b5af0d3b3ec6dd" candidate_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e3989d8c725426da93f7200472bce7b"
I20260618 22:38:45.778312 8160 raft_consensus.cc:3037] T a881dad2cce440f984b5af0d3b3ec6dd P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:45.778343 7878 raft_consensus.cc:3037] T a881dad2cce440f984b5af0d3b3ec6dd P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:45.781124 7878 raft_consensus.cc:2454] T a881dad2cce440f984b5af0d3b3ec6dd P 7e3989d8c725426da93f7200472bce7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a7f01d4d06f4b7eba5cb53475417fd6 in term 1.
I20260618 22:38:45.781137 8160 raft_consensus.cc:2454] T a881dad2cce440f984b5af0d3b3ec6dd P eebd71fa48eb46cd8d0f506ffbff04e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a7f01d4d06f4b7eba5cb53475417fd6 in term 1.
I20260618 22:38:45.781497 7951 leader_election.cc:302] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a7f01d4d06f4b7eba5cb53475417fd6, eebd71fa48eb46cd8d0f506ffbff04e0; no voters:
I20260618 22:38:45.781759 8336 raft_consensus.cc:2781] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:38:45.782125 8336 raft_consensus.cc:686] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 LEADER]: Becoming Leader. State: Replica: 0a7f01d4d06f4b7eba5cb53475417fd6, State: Running, Role: LEADER
I20260618 22:38:45.782375 8336 consensus_queue.cc:237] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:45.784905 7722 catalog_manager.cc:5495] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a7f01d4d06f4b7eba5cb53475417fd6 (127.5.213.194). New cstate: current_term: 1 leader_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } health_report { overall_health: UNKNOWN } } }
I20260618 22:38:45.810688 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:45.805547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54710 (local address 127.5.213.194:37157)
0618 22:38:45.806044 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:45.806048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:45.806066 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:38:45.806112 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:45.806115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:45.806169 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:45.806245 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:38:45.806577 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:45.807329 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:45.809132 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:45.809546 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:45.809641 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:45.810060 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:38:45.810156 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:38:45.810217 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:45.810324 (+ 107us) server_negotiation.cc:299] Negotiation successful
0618 22:38:45.810378 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":268,"threads_started":1}
I20260618 22:38:45.819334 8160 raft_consensus.cc:1261] T a881dad2cce440f984b5af0d3b3ec6dd P eebd71fa48eb46cd8d0f506ffbff04e0 [term 1 FOLLOWER]: Refusing update from remote peer 0a7f01d4d06f4b7eba5cb53475417fd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:38:45.819473 7878 raft_consensus.cc:1261] T a881dad2cce440f984b5af0d3b3ec6dd P 7e3989d8c725426da93f7200472bce7b [term 1 FOLLOWER]: Refusing update from remote peer 0a7f01d4d06f4b7eba5cb53475417fd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:38:45.819869 8337 consensus_queue.cc:1035] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:38:45.820188 8336 consensus_queue.cc:1035] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:38:45.823616 8301 mvcc.cc:205] Tried to move back new op lower bound from 7298344246551629824 to 7298344246406459392. Current Snapshot: MvccSnapshot[applied={T|T < 7298344246551629824}]
I20260618 22:38:45.872445 7721 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:38:45.872641 7721 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:38:45.875201 7721 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d: Sending DeleteTablet for 3 replicas of tablet a881dad2cce440f984b5af0d3b3ec6dd
I20260618 22:38:45.875854 7858 tablet_service.cc:1514] Processing DeleteTablet for tablet a881dad2cce440f984b5af0d3b3ec6dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:38:45 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:46294
I20260618 22:38:45.875912 7997 tablet_service.cc:1514] Processing DeleteTablet for tablet a881dad2cce440f984b5af0d3b3ec6dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:38:45 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:54686
I20260618 22:38:45.876000 8136 tablet_service.cc:1514] Processing DeleteTablet for tablet a881dad2cce440f984b5af0d3b3ec6dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:38:45 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:59582
I20260618 22:38:45.876116 8319 tablet_replica.cc:330] T a881dad2cce440f984b5af0d3b3ec6dd P 7e3989d8c725426da93f7200472bce7b: stopping tablet replica
I20260618 22:38:45.876153 8320 tablet_replica.cc:330] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6: stopping tablet replica
I20260618 22:38:45.876317 8320 raft_consensus.cc:2229] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:38:45.876307 8319 raft_consensus.cc:2229] T a881dad2cce440f984b5af0d3b3ec6dd P 7e3989d8c725426da93f7200472bce7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:38:45.876470 8319 raft_consensus.cc:2258] T a881dad2cce440f984b5af0d3b3ec6dd P 7e3989d8c725426da93f7200472bce7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:38:45.876601 8320 raft_consensus.cc:2258] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:38:45.876708 8321 tablet_replica.cc:330] T a881dad2cce440f984b5af0d3b3ec6dd P eebd71fa48eb46cd8d0f506ffbff04e0: stopping tablet replica
I20260618 22:38:45.876891 8321 raft_consensus.cc:2229] T a881dad2cce440f984b5af0d3b3ec6dd P eebd71fa48eb46cd8d0f506ffbff04e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:38:45.877074 8321 raft_consensus.cc:2258] T a881dad2cce440f984b5af0d3b3ec6dd P eebd71fa48eb46cd8d0f506ffbff04e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:38:45.877030 7721 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260618 22:38:45.877568 8320 ts_tablet_manager.cc:1905] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260618 22:38:45.877601 7721 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:38:45.877941 8319 ts_tablet_manager.cc:1905] T a881dad2cce440f984b5af0d3b3ec6dd P 7e3989d8c725426da93f7200472bce7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:38:45.879225 8321 ts_tablet_manager.cc:1905] T a881dad2cce440f984b5af0d3b3ec6dd P eebd71fa48eb46cd8d0f506ffbff04e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:38:45.882287 8319 ts_tablet_manager.cc:1918] T a881dad2cce440f984b5af0d3b3ec6dd P 7e3989d8c725426da93f7200472bce7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:38:45.882390 8319 log.cc:1196] T a881dad2cce440f984b5af0d3b3ec6dd P 7e3989d8c725426da93f7200472bce7b: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/a881dad2cce440f984b5af0d3b3ec6dd
I20260618 22:38:45.882671 8319 ts_tablet_manager.cc:1939] T a881dad2cce440f984b5af0d3b3ec6dd P 7e3989d8c725426da93f7200472bce7b: Deleting consensus metadata
I20260618 22:38:45.883924 8320 ts_tablet_manager.cc:1918] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:38:45.884059 8320 log.cc:1196] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/a881dad2cce440f984b5af0d3b3ec6dd
I20260618 22:38:45.884418 8320 ts_tablet_manager.cc:1939] T a881dad2cce440f984b5af0d3b3ec6dd P 0a7f01d4d06f4b7eba5cb53475417fd6: Deleting consensus metadata
I20260618 22:38:45.884446 7707 catalog_manager.cc:4841] TS 7e3989d8c725426da93f7200472bce7b (127.5.213.193:33271): tablet a881dad2cce440f984b5af0d3b3ec6dd (table test-table [id=7a55a8c5ee954a5ea0386e49b9005acb]) successfully deleted
I20260618 22:38:45.885198 8321 ts_tablet_manager.cc:1918] T a881dad2cce440f984b5af0d3b3ec6dd P eebd71fa48eb46cd8d0f506ffbff04e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:38:45.885334 8321 log.cc:1196] T a881dad2cce440f984b5af0d3b3ec6dd P eebd71fa48eb46cd8d0f506ffbff04e0: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/a881dad2cce440f984b5af0d3b3ec6dd
I20260618 22:38:45.885339 7708 catalog_manager.cc:4841] TS 0a7f01d4d06f4b7eba5cb53475417fd6 (127.5.213.194:37157): tablet a881dad2cce440f984b5af0d3b3ec6dd (table test-table [id=7a55a8c5ee954a5ea0386e49b9005acb]) successfully deleted
I20260618 22:38:45.885710 8321 ts_tablet_manager.cc:1939] T a881dad2cce440f984b5af0d3b3ec6dd P eebd71fa48eb46cd8d0f506ffbff04e0: Deleting consensus metadata
I20260618 22:38:45.886535 7709 catalog_manager.cc:4841] TS eebd71fa48eb46cd8d0f506ffbff04e0 (127.5.213.195:43811): tablet a881dad2cce440f984b5af0d3b3ec6dd (table test-table [id=7a55a8c5ee954a5ea0386e49b9005acb]) successfully deleted
I20260618 22:38:45.887064 7997 tablet_service.cc:1467] Processing CreateTablet for tablet bbcdd2424973475e9d4764956733e710 (DEFAULT_TABLE table=test-table [id=3108d441190049eca5fac0d07fcb3143]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:45.887264 7858 tablet_service.cc:1467] Processing CreateTablet for tablet bbcdd2424973475e9d4764956733e710 (DEFAULT_TABLE table=test-table [id=3108d441190049eca5fac0d07fcb3143]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:45.887279 8136 tablet_service.cc:1467] Processing CreateTablet for tablet bbcdd2424973475e9d4764956733e710 (DEFAULT_TABLE table=test-table [id=3108d441190049eca5fac0d07fcb3143]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:38:45.887455 7997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbcdd2424973475e9d4764956733e710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:45.887609 8136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbcdd2424973475e9d4764956733e710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:45.887612 7858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbcdd2424973475e9d4764956733e710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:45.891692 8330 tablet_bootstrap.cc:492] T bbcdd2424973475e9d4764956733e710 P 7e3989d8c725426da93f7200472bce7b: Bootstrap starting.
I20260618 22:38:45.893064 8330 tablet_bootstrap.cc:654] T bbcdd2424973475e9d4764956733e710 P 7e3989d8c725426da93f7200472bce7b: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:45.893149 8332 tablet_bootstrap.cc:492] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6: Bootstrap starting.
I20260618 22:38:45.894395 8332 tablet_bootstrap.cc:654] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:45.894765 8330 tablet_bootstrap.cc:492] T bbcdd2424973475e9d4764956733e710 P 7e3989d8c725426da93f7200472bce7b: No bootstrap required, opened a new log
I20260618 22:38:45.894910 8330 ts_tablet_manager.cc:1397] T bbcdd2424973475e9d4764956733e710 P 7e3989d8c725426da93f7200472bce7b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20260618 22:38:45.895637 8331 tablet_bootstrap.cc:492] T bbcdd2424973475e9d4764956733e710 P eebd71fa48eb46cd8d0f506ffbff04e0: Bootstrap starting.
I20260618 22:38:45.895596 8330 raft_consensus.cc:348] T bbcdd2424973475e9d4764956733e710 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:45.895821 8330 raft_consensus.cc:374] T bbcdd2424973475e9d4764956733e710 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:45.895893 8330 raft_consensus.cc:729] T bbcdd2424973475e9d4764956733e710 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e3989d8c725426da93f7200472bce7b, State: Initialized, Role: FOLLOWER
I20260618 22:38:45.896092 8330 consensus_queue.cc:260] T bbcdd2424973475e9d4764956733e710 P 7e3989d8c725426da93f7200472bce7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:45.896389 8332 tablet_bootstrap.cc:492] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6: No bootstrap required, opened a new log
I20260618 22:38:45.896520 8332 ts_tablet_manager.cc:1397] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 22:38:45.896541 8330 ts_tablet_manager.cc:1428] T bbcdd2424973475e9d4764956733e710 P 7e3989d8c725426da93f7200472bce7b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260618 22:38:45.897080 8331 tablet_bootstrap.cc:654] T bbcdd2424973475e9d4764956733e710 P eebd71fa48eb46cd8d0f506ffbff04e0: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:45.897145 8332 raft_consensus.cc:348] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:45.897379 8332 raft_consensus.cc:374] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:45.897445 8332 raft_consensus.cc:729] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a7f01d4d06f4b7eba5cb53475417fd6, State: Initialized, Role: FOLLOWER
I20260618 22:38:45.897696 8332 consensus_queue.cc:260] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:45.898155 8332 ts_tablet_manager.cc:1428] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260618 22:38:45.899396 8331 tablet_bootstrap.cc:492] T bbcdd2424973475e9d4764956733e710 P eebd71fa48eb46cd8d0f506ffbff04e0: No bootstrap required, opened a new log
I20260618 22:38:45.899529 8331 ts_tablet_manager.cc:1397] T bbcdd2424973475e9d4764956733e710 P eebd71fa48eb46cd8d0f506ffbff04e0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260618 22:38:45.900094 8331 raft_consensus.cc:348] T bbcdd2424973475e9d4764956733e710 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:45.900312 8331 raft_consensus.cc:374] T bbcdd2424973475e9d4764956733e710 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:45.900384 8331 raft_consensus.cc:729] T bbcdd2424973475e9d4764956733e710 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebd71fa48eb46cd8d0f506ffbff04e0, State: Initialized, Role: FOLLOWER
I20260618 22:38:45.900596 8331 consensus_queue.cc:260] T bbcdd2424973475e9d4764956733e710 P eebd71fa48eb46cd8d0f506ffbff04e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:45.901131 8331 ts_tablet_manager.cc:1428] T bbcdd2424973475e9d4764956733e710 P eebd71fa48eb46cd8d0f506ffbff04e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260618 22:38:45.939878 8337 raft_consensus.cc:482] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:38:45.940069 8337 raft_consensus.cc:504] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:45.940654 8337 leader_election.cc:288] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e3989d8c725426da93f7200472bce7b (127.5.213.193:33271), eebd71fa48eb46cd8d0f506ffbff04e0 (127.5.213.195:43811)
I20260618 22:38:45.941124 7878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcdd2424973475e9d4764956733e710" candidate_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e3989d8c725426da93f7200472bce7b" is_pre_election: true
I20260618 22:38:45.941140 8160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcdd2424973475e9d4764956733e710" candidate_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" is_pre_election: true
I20260618 22:38:45.941349 8160 raft_consensus.cc:2454] T bbcdd2424973475e9d4764956733e710 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a7f01d4d06f4b7eba5cb53475417fd6 in term 0.
I20260618 22:38:45.941346 7878 raft_consensus.cc:2454] T bbcdd2424973475e9d4764956733e710 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a7f01d4d06f4b7eba5cb53475417fd6 in term 0.
I20260618 22:38:45.941717 7951 leader_election.cc:302] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a7f01d4d06f4b7eba5cb53475417fd6, eebd71fa48eb46cd8d0f506ffbff04e0; no voters:
I20260618 22:38:45.942014 8337 raft_consensus.cc:2781] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:38:45.942107 8337 raft_consensus.cc:482] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:38:45.942211 8337 raft_consensus.cc:3037] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:45.944173 8337 raft_consensus.cc:504] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:45.944619 8337 leader_election.cc:288] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [CANDIDATE]: Term 1 election: Requested vote from peers 7e3989d8c725426da93f7200472bce7b (127.5.213.193:33271), eebd71fa48eb46cd8d0f506ffbff04e0 (127.5.213.195:43811)
I20260618 22:38:45.944969 7878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcdd2424973475e9d4764956733e710" candidate_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e3989d8c725426da93f7200472bce7b"
I20260618 22:38:45.945070 8160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcdd2424973475e9d4764956733e710" candidate_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0"
I20260618 22:38:45.945174 7878 raft_consensus.cc:3037] T bbcdd2424973475e9d4764956733e710 P 7e3989d8c725426da93f7200472bce7b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:45.945246 8160 raft_consensus.cc:3037] T bbcdd2424973475e9d4764956733e710 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:45.947942 7878 raft_consensus.cc:2454] T bbcdd2424973475e9d4764956733e710 P 7e3989d8c725426da93f7200472bce7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a7f01d4d06f4b7eba5cb53475417fd6 in term 1.
I20260618 22:38:45.947959 8160 raft_consensus.cc:2454] T bbcdd2424973475e9d4764956733e710 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a7f01d4d06f4b7eba5cb53475417fd6 in term 1.
I20260618 22:38:45.948323 7949 leader_election.cc:302] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a7f01d4d06f4b7eba5cb53475417fd6, 7e3989d8c725426da93f7200472bce7b; no voters:
I20260618 22:38:45.948556 8337 raft_consensus.cc:2781] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:38:45.948693 8337 raft_consensus.cc:686] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 LEADER]: Becoming Leader. State: Replica: 0a7f01d4d06f4b7eba5cb53475417fd6, State: Running, Role: LEADER
I20260618 22:38:45.948930 8337 consensus_queue.cc:237] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } }
I20260618 22:38:45.951100 7722 catalog_manager.cc:5495] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a7f01d4d06f4b7eba5cb53475417fd6 (127.5.213.194). New cstate: current_term: 1 leader_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a7f01d4d06f4b7eba5cb53475417fd6" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 37157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 } health_report { overall_health: UNKNOWN } } }
I20260618 22:38:45.970803 7878 raft_consensus.cc:1261] T bbcdd2424973475e9d4764956733e710 P 7e3989d8c725426da93f7200472bce7b [term 1 FOLLOWER]: Refusing update from remote peer 0a7f01d4d06f4b7eba5cb53475417fd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:38:45.971300 8337 consensus_queue.cc:1035] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e3989d8c725426da93f7200472bce7b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:38:45.972438 8160 raft_consensus.cc:1261] T bbcdd2424973475e9d4764956733e710 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 1 FOLLOWER]: Refusing update from remote peer 0a7f01d4d06f4b7eba5cb53475417fd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:38:45.972977 8337 consensus_queue.cc:1035] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebd71fa48eb46cd8d0f506ffbff04e0" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260618 22:38:45.993417 8301 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260618 22:38:47.584841 7721 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:38:47.585111 7721 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:38:47.589061 7721 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0f3aeca28c8467dadf0ea56af4c1f6d: Sending DeleteTablet for 3 replicas of tablet bbcdd2424973475e9d4764956733e710
I20260618 22:38:47.589764 7858 tablet_service.cc:1514] Processing DeleteTablet for tablet bbcdd2424973475e9d4764956733e710 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:38:47 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:46294
I20260618 22:38:47.589763 7997 tablet_service.cc:1514] Processing DeleteTablet for tablet bbcdd2424973475e9d4764956733e710 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:38:47 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:54686
I20260618 22:38:47.589880 8136 tablet_service.cc:1514] Processing DeleteTablet for tablet bbcdd2424973475e9d4764956733e710 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:38:47 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:59582
I20260618 22:38:47.590458 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 7793
I20260618 22:38:47.592542 8377 tablet_replica.cc:330] T bbcdd2424973475e9d4764956733e710 P eebd71fa48eb46cd8d0f506ffbff04e0: stopping tablet replica
I20260618 22:38:47.592818 8377 raft_consensus.cc:2229] T bbcdd2424973475e9d4764956733e710 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:38:47.593151 8377 raft_consensus.cc:2258] T bbcdd2424973475e9d4764956733e710 P eebd71fa48eb46cd8d0f506ffbff04e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:38:47.594573 8377 ts_tablet_manager.cc:1905] T bbcdd2424973475e9d4764956733e710 P eebd71fa48eb46cd8d0f506ffbff04e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:38:47.601410 8376 tablet_replica.cc:330] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6: stopping tablet replica
I20260618 22:38:47.601699 8376 raft_consensus.cc:2229] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:38:47.602123 8376 raft_consensus.cc:2258] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:38:47.602559 8375 tablet_replica.cc:330] T bbcdd2424973475e9d4764956733e710 P 7e3989d8c725426da93f7200472bce7b: stopping tablet replica
I20260618 22:38:47.603062 8377 ts_tablet_manager.cc:1918] T bbcdd2424973475e9d4764956733e710 P eebd71fa48eb46cd8d0f506ffbff04e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 22:38:47.603060 8375 raft_consensus.cc:2229] T bbcdd2424973475e9d4764956733e710 P 7e3989d8c725426da93f7200472bce7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:38:47.603272 8377 log.cc:1196] T bbcdd2424973475e9d4764956733e710 P eebd71fa48eb46cd8d0f506ffbff04e0: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/bbcdd2424973475e9d4764956733e710
I20260618 22:38:47.603291 8375 raft_consensus.cc:2258] T bbcdd2424973475e9d4764956733e710 P 7e3989d8c725426da93f7200472bce7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:38:47.603796 8377 ts_tablet_manager.cc:1939] T bbcdd2424973475e9d4764956733e710 P eebd71fa48eb46cd8d0f506ffbff04e0: Deleting consensus metadata
I20260618 22:38:47.603808 8376 ts_tablet_manager.cc:1905] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:38:47.604190 8375 ts_tablet_manager.cc:1905] T bbcdd2424973475e9d4764956733e710 P 7e3989d8c725426da93f7200472bce7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:38:47.604869 7709 catalog_manager.cc:4841] TS eebd71fa48eb46cd8d0f506ffbff04e0 (127.5.213.195:43811): tablet bbcdd2424973475e9d4764956733e710 (table test-table [id=3108d441190049eca5fac0d07fcb3143]) successfully deleted
I20260618 22:38:47.610154 8376 ts_tablet_manager.cc:1918] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 22:38:47.610302 8376 log.cc:1196] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/bbcdd2424973475e9d4764956733e710
I20260618 22:38:47.610369 8375 ts_tablet_manager.cc:1918] T bbcdd2424973475e9d4764956733e710 P 7e3989d8c725426da93f7200472bce7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 22:38:47.610489 8375 log.cc:1196] T bbcdd2424973475e9d4764956733e710 P 7e3989d8c725426da93f7200472bce7b: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/bbcdd2424973475e9d4764956733e710
I20260618 22:38:47.610761 8376 ts_tablet_manager.cc:1939] T bbcdd2424973475e9d4764956733e710 P 0a7f01d4d06f4b7eba5cb53475417fd6: Deleting consensus metadata
I20260618 22:38:47.610895 8375 ts_tablet_manager.cc:1939] T bbcdd2424973475e9d4764956733e710 P 7e3989d8c725426da93f7200472bce7b: Deleting consensus metadata
I20260618 22:38:47.611796 7708 catalog_manager.cc:4841] TS 0a7f01d4d06f4b7eba5cb53475417fd6 (127.5.213.194:37157): tablet bbcdd2424973475e9d4764956733e710 (table test-table [id=3108d441190049eca5fac0d07fcb3143]) successfully deleted
I20260618 22:38:47.611836 7707 catalog_manager.cc:4841] TS 7e3989d8c725426da93f7200472bce7b (127.5.213.193:33271): tablet bbcdd2424973475e9d4764956733e710 (table test-table [id=3108d441190049eca5fac0d07fcb3143]) successfully deleted
Jun 18 22:38:47 dist-test-slave-lnj3 krb5kdc[7677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822325, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:38:47.622607 8379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:47.591310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.5.213.193:33271)
0618 22:38:47.602406 (+ 11096us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:47.602413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:47.604568 (+ 2155us) server_negotiation.cc:407] Connection header received
0618 22:38:47.604627 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:47.604632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:47.604719 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:47.604808 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:47.606261 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:47.607058 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:47.608226 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:47.608675 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:47.608789 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:47.617514 (+ 8725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:47.617530 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:47.617546 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:47.617588 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:47.620114 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:47.620702 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:47.620709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:47.620711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:47.620879 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:47.621207 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:47.621211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:47.621213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:47.621528 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:47.621823 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:47.622081 (+ 258us) server_negotiation.cc:299] Negotiation successful
0618 22:38:47.622277 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10913,"thread_start_us":10782,"threads_started":1}
W20260618 22:38:47.623227 7923 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46314
W20260618 22:38:47.623770 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 7793: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:38:47.623960 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 7793
I20260618 22:38:47.640853 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 7932
Jun 18 22:38:47 dist-test-slave-lnj3 krb5kdc[7677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822325, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.213.194@KRBTEST.COM
I20260618 22:38:47.654680 8380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:47.641547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54720 (local address 127.5.213.194:37157)
0618 22:38:47.642023 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:47.642029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:47.642045 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:38:47.642109 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:47.642113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:47.642179 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:47.642272 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:47.643410 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:47.644191 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:47.645314 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:47.645755 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:47.645851 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:47.649510 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:47.649521 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:47.649568 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:47.649598 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:47.652135 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:47.652724 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:47.652731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:47.652734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:47.652853 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:47.653175 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:47.653178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:47.653180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:47.653509 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:47.653860 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:47.654116 (+ 256us) server_negotiation.cc:299] Negotiation successful
0618 22:38:47.654314 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":245,"threads_started":1}
W20260618 22:38:47.655236 8062 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54720
W20260618 22:38:47.655738 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 7932: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:38:47.655850 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 7932
I20260618 22:38:47.672403 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 8071
Jun 18 22:38:47 dist-test-slave-lnj3 krb5kdc[7677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822325, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.213.195@KRBTEST.COM
I20260618 22:38:47.685930 8381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:47.673094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.5.213.195:43811)
0618 22:38:47.673580 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:47.673586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:47.673603 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:38:47.673657 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:47.673660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:47.673722 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:47.673807 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:47.675052 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:47.675761 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:47.677013 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:47.677424 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:47.677516 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:47.681284 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:47.681297 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:47.681312 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:47.681342 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:47.683804 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:47.684348 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:47.684354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:47.684356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:47.684459 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:47.684753 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:47.684757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:47.684759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:47.685112 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:47.685326 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:47.685527 (+ 201us) server_negotiation.cc:299] Negotiation successful
0618 22:38:47.685685 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":262,"threads_started":1}
W20260618 22:38:47.686517 8201 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59596
W20260618 22:38:47.687074 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 8071: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:38:47.687222 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 8071
I20260618 22:38:47.703590 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 7690
I20260618 22:38:47.704663 7762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:38:47.816151 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 7690
2026-06-18T22:38:47Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8063 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:38:47 dist-test-slave-lnj3 krb5kdc[8385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:38:47 dist-test-slave-lnj3 krb5kdc[8385](info): set up 2 sockets
Jun 18 22:38:47 dist-test-slave-lnj3 krb5kdc[8385](info): commencing operation
krb5kdc: starting...
W20260618 22:38:48.806464 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.007s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:38:48 dist-test-slave-lnj3 krb5kdc[8385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:38:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:38:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:38:48.969307 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:42317
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:40499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:42317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:49.268687 8398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:49.268975 8398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:49.269073 8398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:49.276319 8398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:38:49.276429 8398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:49.276480 8398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:49.276533 8398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:38:49.276575 8398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:38:49.286325 8398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:40499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:42317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:42317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:49.287607 8398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:49.288812 8398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:49.299502 8403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:49.299523 8406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:49.299664 8404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:49.300381 8398 server_base.cc:1034] running on GCE node
I20260618 22:38:49.301137 8398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:49.302515 8398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:49.303740 8398 hybrid_clock.cc:648] HybridClock initialized: now 1781822329303694 us; error 64 us; skew 500 ppm
I20260618 22:38:49.304057 8398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:49 dist-test-slave-lnj3 krb5kdc[8385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822329, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:49.306893 8398 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:38:49.308236 8398 webserver.cc:460] Webserver started at http://127.5.213.254:44735/ using document root <none> and password file <none>
I20260618 22:38:49.308861 8398 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:49.308969 8398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:49.309253 8398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:49.312095 8398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "3edc06f93903484ab137ceee47e09091"
format_stamp: "Formatted at 2026-06-18 22:38:49 on dist-test-slave-lnj3"
server_key: "08136e38304ab421e84b77f9e48cc2c4"
server_key_iv: "709283d6a8bde034bcf0c02b9278f0de"
server_key_version: "encryptionkey@0"
I20260618 22:38:49.312803 8398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "3edc06f93903484ab137ceee47e09091"
format_stamp: "Formatted at 2026-06-18 22:38:49 on dist-test-slave-lnj3"
server_key: "08136e38304ab421e84b77f9e48cc2c4"
server_key_iv: "709283d6a8bde034bcf0c02b9278f0de"
server_key_version: "encryptionkey@0"
I20260618 22:38:49.317631 8398 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260618 22:38:49.321003 8413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:49.322331 8398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:38:49.322469 8398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "3edc06f93903484ab137ceee47e09091"
format_stamp: "Formatted at 2026-06-18 22:38:49 on dist-test-slave-lnj3"
server_key: "08136e38304ab421e84b77f9e48cc2c4"
server_key_iv: "709283d6a8bde034bcf0c02b9278f0de"
server_key_version: "encryptionkey@0"
I20260618 22:38:49.322674 8398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:49.341277 8398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:49.345135 8398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:49.345393 8398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:49.361863 8398 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:42317
I20260618 22:38:49.361892 8464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:42317 every 8 connection(s)
I20260618 22:38:49.363209 8398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:38:49.367002 8465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:49.372643 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 8398
I20260618 22:38:49.372840 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:38:49.373413 5975 external_mini_cluster.cc:1428] Setting key 223944121a609e0bc2615dd3cea6e8ee
I20260618 22:38:49.377063 8465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091: Bootstrap starting.
I20260618 22:38:49.380436 8465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:49.381428 8465 log.cc:824] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091: Log is configured to *not* fsync() on all Append() calls
I20260618 22:38:49.383905 8465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091: No bootstrap required, opened a new log
Jun 18 22:38:49 dist-test-slave-lnj3 krb5kdc[8385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:49.391315 8465 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3edc06f93903484ab137ceee47e09091" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42317 } }
I20260618 22:38:49.391670 8465 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:49.391764 8465 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3edc06f93903484ab137ceee47e09091, State: Initialized, Role: FOLLOWER
I20260618 22:38:49.392324 8465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3edc06f93903484ab137ceee47e09091" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42317 } }
I20260618 22:38:49.392592 8465 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:38:49.392773 8465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:38:49.392964 8465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:49.394549 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:49.375285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58000 (local address 127.5.213.254:42317)
0618 22:38:49.376017 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:49.376031 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:49.376078 (+ 47us) server_negotiation.cc:407] Connection header received
0618 22:38:49.377729 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:49.377751 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:49.378122 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:49.378441 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:49.380116 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:49.381441 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:49.382649 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:49.383301 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:49.383469 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:49.387019 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:49.387027 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:49.387041 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:49.387075 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:49.389988 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:49.390663 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:49.390672 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:49.390681 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:49.390861 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:49.391202 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:49.391206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:49.391208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:49.391794 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:49.392047 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:49.392703 (+ 656us) server_negotiation.cc:299] Negotiation successful
0618 22:38:49.393162 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":313,"threads_started":1}
I20260618 22:38:49.395763 8465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3edc06f93903484ab137ceee47e09091" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42317 } }
I20260618 22:38:49.396374 8465 leader_election.cc:302] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3edc06f93903484ab137ceee47e09091; no voters:
I20260618 22:38:49.396976 8465 leader_election.cc:288] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:38:49.397083 8470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:38:49.398139 8470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091 [term 1 LEADER]: Becoming Leader. State: Replica: 3edc06f93903484ab137ceee47e09091, State: Running, Role: LEADER
I20260618 22:38:49.398609 8470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3edc06f93903484ab137ceee47e09091" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42317 } }
I20260618 22:38:49.398947 8465 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:38:49.401367 8472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3edc06f93903484ab137ceee47e09091. Latest consensus state: current_term: 1 leader_uuid: "3edc06f93903484ab137ceee47e09091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3edc06f93903484ab137ceee47e09091" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42317 } } }
I20260618 22:38:49.401790 8472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091 [sys.catalog]: This master's current role is: LEADER
I20260618 22:38:49.402673 8471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3edc06f93903484ab137ceee47e09091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3edc06f93903484ab137ceee47e09091" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42317 } } }
I20260618 22:38:49.402998 8471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091 [sys.catalog]: This master's current role is: LEADER
I20260618 22:38:49.407552 8478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:38:49.412658 8478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:38:49.421648 8478 catalog_manager.cc:1347] Generated new cluster ID: f993d5fac7c74987ad15418adbe357c1
I20260618 22:38:49.421761 8478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:38:49.436348 8478 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:38:49.437534 8478 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:38:49.448671 8478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3edc06f93903484ab137ceee47e09091: Generated new TSK 0
I20260618 22:38:49.449517 8478 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
I20260618 22:38:49.498274 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:42317
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:40499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:49.803432 8491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:49.803720 8491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:49.803817 8491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:49.811044 8491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:49.811158 8491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:49.811277 8491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:38:49.820950 8491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:40499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:42317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:49.822141 8491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:49.823412 8491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:49.835423 8496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:49.835451 8497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:49.835525 8499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:49.836423 8491 server_base.cc:1034] running on GCE node
I20260618 22:38:49.836966 8491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:49.837925 8491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:49.839112 8491 hybrid_clock.cc:648] HybridClock initialized: now 1781822329839063 us; error 44 us; skew 500 ppm
I20260618 22:38:49.839401 8491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:49 dist-test-slave-lnj3 krb5kdc[8385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822329, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:49.842299 8491 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:38:49.843714 8491 webserver.cc:460] Webserver started at http://127.5.213.193:35611/ using document root <none> and password file <none>
I20260618 22:38:49.844305 8491 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:49.844395 8491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:49.844646 8491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:49.847400 8491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "342cd5967f614f8c9e84ffb3b5045d9c"
format_stamp: "Formatted at 2026-06-18 22:38:49 on dist-test-slave-lnj3"
server_key: "4f940f5b2ba319cffc8037bfff455977"
server_key_iv: "981a7f63006e9001d17d894ed1168905"
server_key_version: "encryptionkey@0"
I20260618 22:38:49.848088 8491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "342cd5967f614f8c9e84ffb3b5045d9c"
format_stamp: "Formatted at 2026-06-18 22:38:49 on dist-test-slave-lnj3"
server_key: "4f940f5b2ba319cffc8037bfff455977"
server_key_iv: "981a7f63006e9001d17d894ed1168905"
server_key_version: "encryptionkey@0"
I20260618 22:38:49.852844 8491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260618 22:38:49.856199 8506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:49.857517 8491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260618 22:38:49.857653 8491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "342cd5967f614f8c9e84ffb3b5045d9c"
format_stamp: "Formatted at 2026-06-18 22:38:49 on dist-test-slave-lnj3"
server_key: "4f940f5b2ba319cffc8037bfff455977"
server_key_iv: "981a7f63006e9001d17d894ed1168905"
server_key_version: "encryptionkey@0"
I20260618 22:38:49.857836 8491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:49.873489 8491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:49.877194 8491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:49.877429 8491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:49.878358 8491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:38:49.879630 8491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:49.879699 8491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:49.879784 8491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:49.879828 8491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:49.907087 8491 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:36819
I20260618 22:38:49.907159 8618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:36819 every 8 connection(s)
I20260618 22:38:49.908385 8491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:38:49.917666 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 8491
I20260618 22:38:49.917860 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:38:49.918406 5975 external_mini_cluster.cc:1428] Setting key 65be2571018933e5d6aa1d95d56f735d
Jun 18 22:38:49 dist-test-slave-lnj3 krb5kdc[8385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822329, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:49.927155 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:49.910745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:60579 (local address 127.5.213.254:42317)
0618 22:38:49.911111 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:49.911115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:49.912071 (+ 956us) server_negotiation.cc:407] Connection header received
0618 22:38:49.913838 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:49.913845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:49.913930 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:49.914023 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:49.915505 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:49.916201 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:49.917705 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:49.918188 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:49.918293 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:49.921828 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:49.921833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:49.921836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:49.921866 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:49.924115 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:49.924835 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:49.924839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:49.924841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:49.924954 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:49.925341 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:49.925345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:49.925347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:49.925652 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:49.925796 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:49.926504 (+ 708us) server_negotiation.cc:299] Negotiation successful
0618 22:38:49.926748 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":181,"threads_started":1}
I20260618 22:38:49.928079 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:49.911248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:42317 (local address 127.5.213.193:60579)
0618 22:38:49.911923 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:49.911966 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:49.913567 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:49.914258 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:49.914266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:49.914806 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:49.915351 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:49.915367 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:49.916349 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:49.916353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:49.917576 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:49.917587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:49.918390 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:49.918397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:49.918651 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:49.919248 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:49.919270 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:49.921625 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:49.924237 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:49.924246 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:49.924272 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:49.924707 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:49.925053 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:49.925056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:49.925058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:49.925226 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:49.925764 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:49.925771 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:49.926104 (+ 333us) client_negotiation.cc:769] Sending connection context
0618 22:38:49.926399 (+ 295us) client_negotiation.cc:240] Negotiation successful
0618 22:38:49.927014 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":246,"threads_started":1}
I20260618 22:38:49.929441 8619 heartbeater.cc:347] Connected to a master server at 127.5.213.254:42317
I20260618 22:38:49.929751 8619 heartbeater.cc:464] Registering TS with master...
I20260618 22:38:49.930584 8619 heartbeater.cc:511] Master 127.5.213.254:42317 requested a full tablet report, sending...
I20260618 22:38:49.932642 8430 ts_manager.cc:194] Registered new tserver with Master: 342cd5967f614f8c9e84ffb3b5045d9c (127.5.213.193:36819)
I20260618 22:38:49.934020 8430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:60579
WARNING: no policy specified for kudu/127.5.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
I20260618 22:38:49.955730 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:42317
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:40499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:50.254428 8625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:50.254739 8625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:50.254887 8625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:50.261930 8625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:50.262029 8625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:50.262166 8625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:38:50.271574 8625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:40499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:42317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:50.272766 8625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:50.274016 8625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:50.285943 8630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:50.286000 8633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:50.285967 8631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:50.286615 8625 server_base.cc:1034] running on GCE node
I20260618 22:38:50.287197 8625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:50.288012 8625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:50.289207 8625 hybrid_clock.cc:648] HybridClock initialized: now 1781822330289152 us; error 39 us; skew 500 ppm
I20260618 22:38:50.289479 8625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:50 dist-test-slave-lnj3 krb5kdc[8385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822330, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:50.292337 8625 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:38:50.293604 8625 webserver.cc:460] Webserver started at http://127.5.213.194:42519/ using document root <none> and password file <none>
I20260618 22:38:50.294230 8625 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:50.294337 8625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:50.294602 8625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:50.297322 8625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "1a05ae19fabc4bb3b356d66c0e8fa785"
format_stamp: "Formatted at 2026-06-18 22:38:50 on dist-test-slave-lnj3"
server_key: "ced1e8b80a8431d312dcf6accc9bea03"
server_key_iv: "d677bb6ca33c42dafaef2c95d887b3ca"
server_key_version: "encryptionkey@0"
I20260618 22:38:50.298027 8625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "1a05ae19fabc4bb3b356d66c0e8fa785"
format_stamp: "Formatted at 2026-06-18 22:38:50 on dist-test-slave-lnj3"
server_key: "ced1e8b80a8431d312dcf6accc9bea03"
server_key_iv: "d677bb6ca33c42dafaef2c95d887b3ca"
server_key_version: "encryptionkey@0"
I20260618 22:38:50.302944 8625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260618 22:38:50.306074 8640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:50.307415 8625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:38:50.307548 8625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "1a05ae19fabc4bb3b356d66c0e8fa785"
format_stamp: "Formatted at 2026-06-18 22:38:50 on dist-test-slave-lnj3"
server_key: "ced1e8b80a8431d312dcf6accc9bea03"
server_key_iv: "d677bb6ca33c42dafaef2c95d887b3ca"
server_key_version: "encryptionkey@0"
I20260618 22:38:50.307734 8625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:50.322253 8625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:50.326098 8625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:50.326334 8625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:50.327243 8625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:38:50.328452 8625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:50.328521 8625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:50.328629 8625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:50.328683 8625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:50.357405 8625 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:34003
I20260618 22:38:50.357446 8752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:34003 every 8 connection(s)
I20260618 22:38:50.358608 8625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:38:50.360100 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 8625
I20260618 22:38:50.360281 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:38:50.360805 5975 external_mini_cluster.cc:1428] Setting key e4fbc29220ae1bf938f6dc86e6b1c029
Jun 18 22:38:50 dist-test-slave-lnj3 krb5kdc[8385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822330, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:50.379381 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:50.361199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:33507 (local address 127.5.213.254:42317)
0618 22:38:50.361384 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:50.361390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:50.362768 (+ 1378us) server_negotiation.cc:407] Connection header received
0618 22:38:50.364514 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:50.364519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:50.364622 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:50.364701 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:50.366097 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:50.366886 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:50.368435 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:50.369080 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:50.369196 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:50.372639 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:50.372646 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:50.372651 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:50.372686 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:50.376050 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:50.376905 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:50.376910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:50.376914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:50.377080 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:50.377558 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:50.377568 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:50.377571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:50.377906 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:50.378061 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:50.378842 (+ 781us) server_negotiation.cc:299] Negotiation successful
0618 22:38:50.379055 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
WARNING: no policy specified for kudu/127.5.213.195@KRBTEST.COM; defaulting to no policy
I20260618 22:38:50.380453 8755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:50.361778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:42317 (local address 127.5.213.194:33507)
0618 22:38:50.362601 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:50.362679 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:50.364277 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:50.364934 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:50.364941 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:50.365424 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:50.365944 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:50.365961 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:50.367031 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:50.367036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:50.368275 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:50.368290 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:50.369283 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:50.369288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:50.369481 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:50.370083 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:50.370104 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:50.372428 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:50.376213 (+ 3785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:50.376221 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:50.376237 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:50.376783 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:50.377211 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:50.377215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:50.377219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:50.377433 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:50.378033 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:50.378041 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:50.378437 (+ 396us) client_negotiation.cc:769] Sending connection context
0618 22:38:50.378757 (+ 320us) client_negotiation.cc:240] Negotiation successful
0618 22:38:50.379233 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":363,"threads_started":1}
I20260618 22:38:50.381843 8753 heartbeater.cc:347] Connected to a master server at 127.5.213.254:42317
I20260618 22:38:50.382138 8753 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.195@KRBTEST.COM" created.
I20260618 22:38:50.382974 8753 heartbeater.cc:511] Master 127.5.213.254:42317 requested a full tablet report, sending...
I20260618 22:38:50.384472 8430 ts_manager.cc:194] Registered new tserver with Master: 1a05ae19fabc4bb3b356d66c0e8fa785 (127.5.213.194:34003)
I20260618 22:38:50.385505 8430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:33507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
I20260618 22:38:50.403337 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:42317
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:40499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:50.710901 8758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:50.711181 8758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:50.711274 8758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:50.718454 8758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:50.718564 8758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:50.718739 8758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:38:50.728415 8758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:40499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:42317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:50.729624 8758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:50.730880 8758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:50.745190 8766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:50.745429 8764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:50.745409 8763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:50.746207 8758 server_base.cc:1034] running on GCE node
I20260618 22:38:50.746876 8758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:50.747848 8758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:50.749032 8758 hybrid_clock.cc:648] HybridClock initialized: now 1781822330748972 us; error 53 us; skew 500 ppm
I20260618 22:38:50.749321 8758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:50 dist-test-slave-lnj3 krb5kdc[8385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822330, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:50.752379 8758 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:38:50.753767 8758 webserver.cc:460] Webserver started at http://127.5.213.195:45389/ using document root <none> and password file <none>
I20260618 22:38:50.754408 8758 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:50.754513 8758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:50.754833 8758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:50.757615 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "6ff9d6c4d3b842acb8dc4a0f07aa99bb"
format_stamp: "Formatted at 2026-06-18 22:38:50 on dist-test-slave-lnj3"
server_key: "7e7e1f99f42867c07a7b0c0096968d56"
server_key_iv: "7aabc54a3e545b5ad57a1ea3b1c0565f"
server_key_version: "encryptionkey@0"
I20260618 22:38:50.758358 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "6ff9d6c4d3b842acb8dc4a0f07aa99bb"
format_stamp: "Formatted at 2026-06-18 22:38:50 on dist-test-slave-lnj3"
server_key: "7e7e1f99f42867c07a7b0c0096968d56"
server_key_iv: "7aabc54a3e545b5ad57a1ea3b1c0565f"
server_key_version: "encryptionkey@0"
I20260618 22:38:50.763353 8758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260618 22:38:50.766541 8773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:50.767886 8758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260618 22:38:50.768024 8758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "6ff9d6c4d3b842acb8dc4a0f07aa99bb"
format_stamp: "Formatted at 2026-06-18 22:38:50 on dist-test-slave-lnj3"
server_key: "7e7e1f99f42867c07a7b0c0096968d56"
server_key_iv: "7aabc54a3e545b5ad57a1ea3b1c0565f"
server_key_version: "encryptionkey@0"
I20260618 22:38:50.768210 8758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:50.797896 8758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:50.802181 8758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:50.802443 8758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:50.803544 8758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:38:50.804870 8758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:50.804940 8758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:50.805037 8758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:50.805086 8758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:50.834757 8758 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:36879
I20260618 22:38:50.834822 8885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:36879 every 8 connection(s)
I20260618 22:38:50.836025 8758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:38:50.839594 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 8758
I20260618 22:38:50.839779 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:38:50.840286 5975 external_mini_cluster.cc:1428] Setting key 545435b3de024dea5051262abcbca77c
Jun 18 22:38:50 dist-test-slave-lnj3 krb5kdc[8385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822330, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:50.855053 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:50.838368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:34319 (local address 127.5.213.254:42317)
0618 22:38:50.838577 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:50.838583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:50.839772 (+ 1189us) server_negotiation.cc:407] Connection header received
0618 22:38:50.841795 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:50.841800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:50.841886 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:50.841968 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:50.843573 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:50.844284 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:50.845757 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:50.846144 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:50.846231 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:50.849593 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:50.849598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:50.849601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:50.849639 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:50.852079 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:50.852766 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:50.852769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:50.852772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:50.852875 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:50.853254 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:50.853258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:50.853262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:50.853598 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:50.853716 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:50.854464 (+ 748us) server_negotiation.cc:299] Negotiation successful
0618 22:38:50.854736 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260618 22:38:50.856010 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:50.838961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:42317 (local address 127.5.213.195:34319)
0618 22:38:50.839622 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:50.839659 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:50.841547 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:50.842216 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:50.842224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:50.842784 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:50.843430 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:50.843449 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:50.844402 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:50.844406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:50.845613 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:50.845625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:50.846346 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:50.846351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:50.846593 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:50.847254 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:50.847277 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:50.849384 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:50.852219 (+ 2835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:50.852228 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:50.852241 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:50.852667 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:50.852952 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:50.852955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:50.852957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:50.853170 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:50.853758 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:50.853765 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:50.854091 (+ 326us) client_negotiation.cc:769] Sending connection context
0618 22:38:50.854353 (+ 262us) client_negotiation.cc:240] Negotiation successful
0618 22:38:50.854823 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":272,"threads_started":1}
I20260618 22:38:50.857187 8886 heartbeater.cc:347] Connected to a master server at 127.5.213.254:42317
I20260618 22:38:50.857457 8886 heartbeater.cc:464] Registering TS with master...
I20260618 22:38:50.858284 8886 heartbeater.cc:511] Master 127.5.213.254:42317 requested a full tablet report, sending...
I20260618 22:38:50.859609 8430 ts_manager.cc:194] Registered new tserver with Master: 6ff9d6c4d3b842acb8dc4a0f07aa99bb (127.5.213.195:36879)
I20260618 22:38:50.860385 8430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:34319
I20260618 22:38:50.867405 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260618 22:38:50.894048 8621 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:38:50.891652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49636 (local address 127.5.213.254:42317)
0618 22:38:50.891870 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:50.891876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:50.892233 (+ 357us) server_negotiation.cc:407] Connection header received
0618 22:38:50.892428 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:50.892435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:50.892502 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:50.892587 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:50.893800 (+ 1213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49636: BlockingRecv error: recv got EOF from 127.0.0.1:49636 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:38:50.896961 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 8491
W20260618 22:38:50.899564 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 8491: Not authorized: Client connection negotiation failed: client connection to 127.5.213.193:36819: server requires authentication, but client does not have Kerberos credentials available
I20260618 22:38:50.899708 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 8491
I20260618 22:38:50.911648 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 8625
W20260618 22:38:50.914129 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 8625: Not authorized: Client connection negotiation failed: client connection to 127.5.213.194:34003: server requires authentication, but client does not have Kerberos credentials available
I20260618 22:38:50.914273 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 8625
I20260618 22:38:50.925800 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 8758
W20260618 22:38:50.928511 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 8758: Not authorized: Client connection negotiation failed: client connection to 127.5.213.195:36879: server requires authentication, but client does not have Kerberos credentials available
I20260618 22:38:50.928639 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 8758
I20260618 22:38:50.940011 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 8398
I20260618 22:38:50.941103 8460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:38:51.043708 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 8398
2026-06-18T22:38:51Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3225 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:38:51 dist-test-slave-lnj3 krb5kdc[8903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:38:51 dist-test-slave-lnj3 krb5kdc[8903](info): set up 2 sockets
Jun 18 22:38:51 dist-test-slave-lnj3 krb5kdc[8903](info): commencing operation
krb5kdc: starting...
W20260618 22:38:52.038720 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.004s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:38:52 dist-test-slave-lnj3 krb5kdc[8903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:38:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:38:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:38:52.207109 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:36777
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:36777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:52.511376 8916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:52.511771 8916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:52.511916 8916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:52.520067 8916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:38:52.520174 8916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:52.520226 8916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:52.520280 8916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:38:52.520326 8916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:38:52.529881 8916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:36777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:36777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:52.531134 8916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:52.532330 8916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:52.543159 8921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:52.543226 8922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:52.543205 8924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:52.544096 8916 server_base.cc:1034] running on GCE node
I20260618 22:38:52.544852 8916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:52.546202 8916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:52.547391 8916 hybrid_clock.cc:648] HybridClock initialized: now 1781822332547334 us; error 52 us; skew 500 ppm
I20260618 22:38:52.547710 8916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:52 dist-test-slave-lnj3 krb5kdc[8903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822332, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:52.550566 8916 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:38:52.552019 8916 webserver.cc:460] Webserver started at http://127.5.213.254:42767/ using document root <none> and password file <none>
I20260618 22:38:52.552670 8916 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:52.552822 8916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:52.553092 8916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:52.555956 8916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "b794758533df479da4935a6d779fc835"
format_stamp: "Formatted at 2026-06-18 22:38:52 on dist-test-slave-lnj3"
server_key: "b88734cbdb7ff6315421875a76f22792"
server_key_iv: "931908effa84c8add0fb7147dc586855"
server_key_version: "encryptionkey@0"
I20260618 22:38:52.556720 8916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "b794758533df479da4935a6d779fc835"
format_stamp: "Formatted at 2026-06-18 22:38:52 on dist-test-slave-lnj3"
server_key: "b88734cbdb7ff6315421875a76f22792"
server_key_iv: "931908effa84c8add0fb7147dc586855"
server_key_version: "encryptionkey@0"
I20260618 22:38:52.561616 8916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260618 22:38:52.564908 8931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:52.566177 8916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:38:52.566324 8916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "b794758533df479da4935a6d779fc835"
format_stamp: "Formatted at 2026-06-18 22:38:52 on dist-test-slave-lnj3"
server_key: "b88734cbdb7ff6315421875a76f22792"
server_key_iv: "931908effa84c8add0fb7147dc586855"
server_key_version: "encryptionkey@0"
I20260618 22:38:52.566496 8916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:52.602887 8916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:52.606922 8916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:52.607191 8916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:52.623111 8916 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:36777
I20260618 22:38:52.623139 8982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:36777 every 8 connection(s)
I20260618 22:38:52.624394 8916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:38:52.627774 8983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:52.631412 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 8916
I20260618 22:38:52.631628 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:38:52.632220 5975 external_mini_cluster.cc:1428] Setting key 92ad1ee1f155dc1b7e0bad705cd80db8
I20260618 22:38:52.639014 8983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835: Bootstrap starting.
I20260618 22:38:52.642320 8983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:52.643457 8983 log.cc:824] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835: Log is configured to *not* fsync() on all Append() calls
I20260618 22:38:52.645923 8983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835: No bootstrap required, opened a new log
Jun 18 22:38:52 dist-test-slave-lnj3 krb5kdc[8903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:52.653219 8983 raft_consensus.cc:348] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b794758533df479da4935a6d779fc835" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36777 } }
I20260618 22:38:52.653560 8983 raft_consensus.cc:374] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:52.653672 8983 raft_consensus.cc:729] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b794758533df479da4935a6d779fc835, State: Initialized, Role: FOLLOWER
I20260618 22:38:52.654286 8983 consensus_queue.cc:260] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b794758533df479da4935a6d779fc835" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36777 } }
I20260618 22:38:52.654560 8983 raft_consensus.cc:388] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:38:52.654728 8983 raft_consensus.cc:482] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:38:52.654901 8983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:52.657693 8983 raft_consensus.cc:504] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b794758533df479da4935a6d779fc835" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36777 } }
I20260618 22:38:52.658231 8983 leader_election.cc:302] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b794758533df479da4935a6d779fc835; no voters:
I20260618 22:38:52.658849 8983 leader_election.cc:288] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:38:52.658958 8988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:38:52.660343 8986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:52.633848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42918 (local address 127.5.213.254:36777)
0618 22:38:52.634621 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:52.634657 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:52.634716 (+ 59us) server_negotiation.cc:407] Connection header received
0618 22:38:52.637356 (+ 2640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:52.637383 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:52.637852 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:52.638251 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:52.639519 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:52.641423 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:52.642711 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:52.643683 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:52.643849 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:52.652342 (+ 8493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:52.652350 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:52.652364 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:52.652403 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:52.655826 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:52.656487 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:52.656495 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:52.656503 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:52.656709 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:52.656994 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:52.656999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:52.657003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:52.657654 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:52.657923 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:52.658318 (+ 395us) server_negotiation.cc:299] Negotiation successful
0618 22:38:52.658989 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":322,"threads_started":1}
I20260618 22:38:52.660885 8983 sys_catalog.cc:564] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:38:52.661942 8988 raft_consensus.cc:686] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835 [term 1 LEADER]: Becoming Leader. State: Replica: b794758533df479da4935a6d779fc835, State: Running, Role: LEADER
I20260618 22:38:52.662606 8988 consensus_queue.cc:237] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b794758533df479da4935a6d779fc835" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36777 } }
I20260618 22:38:52.664800 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:38:52.665328 8990 sys_catalog.cc:455] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b794758533df479da4935a6d779fc835. Latest consensus state: current_term: 1 leader_uuid: "b794758533df479da4935a6d779fc835" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b794758533df479da4935a6d779fc835" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36777 } } }
I20260618 22:38:52.665630 8990 sys_catalog.cc:458] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835 [sys.catalog]: This master's current role is: LEADER
I20260618 22:38:52.666066 8989 sys_catalog.cc:455] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b794758533df479da4935a6d779fc835" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b794758533df479da4935a6d779fc835" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36777 } } }
I20260618 22:38:52.666327 8989 sys_catalog.cc:458] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835 [sys.catalog]: This master's current role is: LEADER
I20260618 22:38:52.666489 8995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:38:52.677408 8995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:38:52.693809 8995 catalog_manager.cc:1347] Generated new cluster ID: f695909129164c5c9228168baf4a9349
I20260618 22:38:52.694015 8995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:38:52.712489 8986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:52.703958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42928 (local address 127.5.213.254:36777)
0618 22:38:52.704136 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:52.704142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:52.704677 (+ 535us) server_negotiation.cc:407] Connection header received
0618 22:38:52.704857 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:52.704862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:52.704941 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:52.705051 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:52.706183 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:52.707259 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:52.709079 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:52.709691 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:52.709799 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:52.710964 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:52.710971 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:52.710976 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:38:52.711748 (+ 772us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:52.711815 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:38:52.711848 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:52.711941 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:52.712084 (+ 143us) server_negotiation.cc:299] Negotiation successful
0618 22:38:52.712173 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:38:52.719596 8995 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:38:52.721381 8995 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:38:52.731238 8995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b794758533df479da4935a6d779fc835: Generated new TSK 0
I20260618 22:38:52.732236 8995 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:38:52.741143 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 8916
I20260618 22:38:52.742276 8978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:38:52.829753 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 8916
2026-06-18T22:38:52Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1781 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:38:52 dist-test-slave-lnj3 krb5kdc[9017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:38:52 dist-test-slave-lnj3 krb5kdc[9017](info): set up 2 sockets
Jun 18 22:38:52 dist-test-slave-lnj3 krb5kdc[9017](info): commencing operation
krb5kdc: starting...
W20260618 22:38:53.814733 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:38:53 dist-test-slave-lnj3 krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:38:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:38:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:38:53.978528 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:41515
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:41515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:54.279155 9030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:54.279453 9030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:54.279582 9030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:54.286563 9030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:38:54.286695 9030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:54.286768 9030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:54.286850 9030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:38:54.286906 9030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:38:54.296247 9030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:41515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:41515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:54.297492 9030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:54.298727 9030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:54.309755 9035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:54.309798 9038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:54.309834 9036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:54.310498 9030 server_base.cc:1034] running on GCE node
I20260618 22:38:54.311223 9030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:54.312474 9030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:54.313665 9030 hybrid_clock.cc:648] HybridClock initialized: now 1781822334313607 us; error 50 us; skew 500 ppm
I20260618 22:38:54.313956 9030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:54 dist-test-slave-lnj3 krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822334, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:54.316815 9030 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:38:54.318097 9030 webserver.cc:460] Webserver started at http://127.5.213.254:44255/ using document root <none> and password file <none>
I20260618 22:38:54.318722 9030 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:54.318819 9030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:54.319061 9030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:54.321772 9030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "75403beed4ba4c60b7e753733f6364f8"
format_stamp: "Formatted at 2026-06-18 22:38:54 on dist-test-slave-lnj3"
server_key: "34aef0331d31eb3f00bdc559b74106ad"
server_key_iv: "21c587efb40acbb7d21ad663870ceb2f"
server_key_version: "encryptionkey@0"
I20260618 22:38:54.322422 9030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "75403beed4ba4c60b7e753733f6364f8"
format_stamp: "Formatted at 2026-06-18 22:38:54 on dist-test-slave-lnj3"
server_key: "34aef0331d31eb3f00bdc559b74106ad"
server_key_iv: "21c587efb40acbb7d21ad663870ceb2f"
server_key_version: "encryptionkey@0"
I20260618 22:38:54.327092 9030 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260618 22:38:54.330341 9045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:54.331578 9030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260618 22:38:54.331718 9030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "75403beed4ba4c60b7e753733f6364f8"
format_stamp: "Formatted at 2026-06-18 22:38:54 on dist-test-slave-lnj3"
server_key: "34aef0331d31eb3f00bdc559b74106ad"
server_key_iv: "21c587efb40acbb7d21ad663870ceb2f"
server_key_version: "encryptionkey@0"
I20260618 22:38:54.331879 9030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:54.368798 9030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:54.373034 9030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:54.373294 9030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:54.389202 9030 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:41515
I20260618 22:38:54.389221 9096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:41515 every 8 connection(s)
I20260618 22:38:54.390519 9030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:38:54.394050 9097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:54.395088 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 9030
I20260618 22:38:54.395258 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:38:54.395838 5975 external_mini_cluster.cc:1428] Setting key 1e84da19371bc1152a97ef739d6b2c87
I20260618 22:38:54.407369 9097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8: Bootstrap starting.
Jun 18 22:38:54 dist-test-slave-lnj3 krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:54.410180 9097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:54.411482 9097 log.cc:824] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8: Log is configured to *not* fsync() on all Append() calls
I20260618 22:38:54.414515 9097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8: No bootstrap required, opened a new log
I20260618 22:38:54.417675 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:54.397707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44532 (local address 127.5.213.254:41515)
0618 22:38:54.398447 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:54.398462 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:54.398519 (+ 57us) server_negotiation.cc:407] Connection header received
0618 22:38:54.400207 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:54.400236 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:54.400742 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:54.401168 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:54.402346 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:54.404226 (+ 1880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:54.405412 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:54.406480 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:54.406678 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:54.410107 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:54.410117 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:54.410135 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:54.410181 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:54.413626 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:54.414195 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:54.414203 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:54.414212 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:54.414406 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:54.414761 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:54.414766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:54.414769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:54.415432 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:54.415711 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:54.415958 (+ 247us) server_negotiation.cc:299] Negotiation successful
0618 22:38:54.416388 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":295,"threads_started":1}
I20260618 22:38:54.423062 9097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75403beed4ba4c60b7e753733f6364f8" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 41515 } }
I20260618 22:38:54.423363 9097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:54.423431 9097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75403beed4ba4c60b7e753733f6364f8, State: Initialized, Role: FOLLOWER
I20260618 22:38:54.423949 9097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75403beed4ba4c60b7e753733f6364f8" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 41515 } }
I20260618 22:38:54.424149 9097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:38:54.424239 9097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:38:54.424346 9097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:54.426504 9097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75403beed4ba4c60b7e753733f6364f8" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 41515 } }
I20260618 22:38:54.426981 9097 leader_election.cc:302] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75403beed4ba4c60b7e753733f6364f8; no voters:
I20260618 22:38:54.427479 9097 leader_election.cc:288] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:38:54.427584 9102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:38:54.428200 9102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8 [term 1 LEADER]: Becoming Leader. State: Replica: 75403beed4ba4c60b7e753733f6364f8, State: Running, Role: LEADER
I20260618 22:38:54.428699 9102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75403beed4ba4c60b7e753733f6364f8" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 41515 } }
I20260618 22:38:54.429467 9097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:38:54.431143 9103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75403beed4ba4c60b7e753733f6364f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75403beed4ba4c60b7e753733f6364f8" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 41515 } } }
I20260618 22:38:54.431433 9103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8 [sys.catalog]: This master's current role is: LEADER
I20260618 22:38:54.432178 9108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:38:54.432039 9104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75403beed4ba4c60b7e753733f6364f8. Latest consensus state: current_term: 1 leader_uuid: "75403beed4ba4c60b7e753733f6364f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75403beed4ba4c60b7e753733f6364f8" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 41515 } } }
I20260618 22:38:54.433367 9104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8 [sys.catalog]: This master's current role is: LEADER
I20260618 22:38:54.436741 9108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:38:54.444417 9108 catalog_manager.cc:1347] Generated new cluster ID: c9cfbc9ff0a64aa39a73793c40e8f281
I20260618 22:38:54.444556 9108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:38:54.454823 9108 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:38:54.456002 9108 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:38:54.461459 9108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75403beed4ba4c60b7e753733f6364f8: Generated new TSK 0
I20260618 22:38:54.462126 9108 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
I20260618 22:38:54.503477 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:41515
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:54.809912 9123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:54.810185 9123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:54.810302 9123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:54.817379 9123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:54.817489 9123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:54.817656 9123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:38:54.827129 9123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:41515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:54.828323 9123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:54.829620 9123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:54.842029 9131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:54.842085 9128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:54.842101 9129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:54.843019 9123 server_base.cc:1034] running on GCE node
I20260618 22:38:54.843585 9123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:54.844455 9123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:54.845649 9123 hybrid_clock.cc:648] HybridClock initialized: now 1781822334845597 us; error 38 us; skew 500 ppm
I20260618 22:38:54.845965 9123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:54 dist-test-slave-lnj3 krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822334, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:54.848948 9123 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:38:54.850250 9123 webserver.cc:460] Webserver started at http://127.5.213.193:42205/ using document root <none> and password file <none>
I20260618 22:38:54.850907 9123 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:54.851001 9123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:54.851240 9123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:54.853928 9123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "812f7a649d714b55ac4c0f7150c26887"
format_stamp: "Formatted at 2026-06-18 22:38:54 on dist-test-slave-lnj3"
server_key: "b071afc9e3ffecafe62fee5466678ce2"
server_key_iv: "1f31e3b3917a9abb84965b36d4e5f893"
server_key_version: "encryptionkey@0"
I20260618 22:38:54.854620 9123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "812f7a649d714b55ac4c0f7150c26887"
format_stamp: "Formatted at 2026-06-18 22:38:54 on dist-test-slave-lnj3"
server_key: "b071afc9e3ffecafe62fee5466678ce2"
server_key_iv: "1f31e3b3917a9abb84965b36d4e5f893"
server_key_version: "encryptionkey@0"
I20260618 22:38:54.859421 9123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260618 22:38:54.862671 9138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:54.863947 9123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260618 22:38:54.864071 9123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "812f7a649d714b55ac4c0f7150c26887"
format_stamp: "Formatted at 2026-06-18 22:38:54 on dist-test-slave-lnj3"
server_key: "b071afc9e3ffecafe62fee5466678ce2"
server_key_iv: "1f31e3b3917a9abb84965b36d4e5f893"
server_key_version: "encryptionkey@0"
I20260618 22:38:54.864224 9123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:54.877700 9123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:54.881652 9123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:54.881894 9123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:54.882740 9123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:38:54.883971 9123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:54.884042 9123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:54.884135 9123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:54.884181 9123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:54.912603 9123 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:36047
I20260618 22:38:54.912638 9250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:36047 every 8 connection(s)
I20260618 22:38:54.913921 9123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:38:54.919026 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 9123
I20260618 22:38:54.919188 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:38:54.919744 5975 external_mini_cluster.cc:1428] Setting key 9a5b85e3c9d5c685cc05c47e4c4da6c8
Jun 18 22:38:54 dist-test-slave-lnj3 krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822334, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:54.932356 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:54.916327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:57617 (local address 127.5.213.254:41515)
0618 22:38:54.916530 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:54.916536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:54.917560 (+ 1024us) server_negotiation.cc:407] Connection header received
0618 22:38:54.919334 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:54.919339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:54.919419 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:54.919509 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:54.921078 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:54.921932 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:54.923455 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:54.923861 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:54.923970 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:54.927120 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:54.927125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:54.927128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:54.927156 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:54.929484 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:54.930126 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:54.930132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:54.930135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:54.930297 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:54.930775 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:54.930781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:54.930784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:54.931045 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:54.931177 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:54.931822 (+ 645us) server_negotiation.cc:299] Negotiation successful
0618 22:38:54.932027 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260618 22:38:54.933445 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:54.916765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:41515 (local address 127.5.213.193:57617)
0618 22:38:54.917405 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:54.917453 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:54.919081 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:54.919742 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:54.919750 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:54.920296 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:54.920932 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:54.920950 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:54.922073 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:54.922077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:54.923332 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:54.923343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:54.924062 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:54.924069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:54.924281 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:54.924872 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:54.924895 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:54.926918 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:54.929607 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:54.929617 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:54.929633 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:54.930016 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:54.930389 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:54.930415 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:54.930419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:54.930662 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:54.931156 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:54.931164 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:54.931491 (+ 327us) client_negotiation.cc:769] Sending connection context
0618 22:38:54.931721 (+ 230us) client_negotiation.cc:240] Negotiation successful
0618 22:38:54.932233 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":240,"threads_started":1}
WARNING: no policy specified for kudu/127.5.213.194@KRBTEST.COM; defaulting to no policy
I20260618 22:38:54.934971 9251 heartbeater.cc:347] Connected to a master server at 127.5.213.254:41515
I20260618 22:38:54.935256 9251 heartbeater.cc:464] Registering TS with master...
I20260618 22:38:54.935990 9251 heartbeater.cc:511] Master 127.5.213.254:41515 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.194@KRBTEST.COM" created.
I20260618 22:38:54.938077 9062 ts_manager.cc:194] Registered new tserver with Master: 812f7a649d714b55ac4c0f7150c26887 (127.5.213.193:36047)
I20260618 22:38:54.939523 9062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:57617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
I20260618 22:38:54.957748 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:41515
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:55.260460 9256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:55.260738 9256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:55.260871 9256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:55.268051 9256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:55.268177 9256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:55.268339 9256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:38:55.278347 9256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:41515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:55.279633 9256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:55.280879 9256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:55.293042 9261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:55.293102 9262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:55.293081 9264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:55.293641 9256 server_base.cc:1034] running on GCE node
I20260618 22:38:55.294265 9256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:55.295225 9256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:55.296435 9256 hybrid_clock.cc:648] HybridClock initialized: now 1781822335296381 us; error 47 us; skew 500 ppm
I20260618 22:38:55.296736 9256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:55 dist-test-slave-lnj3 krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822335, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:55.299682 9256 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:38:55.301081 9256 webserver.cc:460] Webserver started at http://127.5.213.194:38701/ using document root <none> and password file <none>
I20260618 22:38:55.301756 9256 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:55.301869 9256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:55.302150 9256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:55.304997 9256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "e2155b2dbd2d4c82aebf8e464b56b5e5"
format_stamp: "Formatted at 2026-06-18 22:38:55 on dist-test-slave-lnj3"
server_key: "893cdd3cd460b95cc8f8bc66d3863178"
server_key_iv: "9a4b016230961cd62faf38e1e53dfe36"
server_key_version: "encryptionkey@0"
I20260618 22:38:55.305725 9256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "e2155b2dbd2d4c82aebf8e464b56b5e5"
format_stamp: "Formatted at 2026-06-18 22:38:55 on dist-test-slave-lnj3"
server_key: "893cdd3cd460b95cc8f8bc66d3863178"
server_key_iv: "9a4b016230961cd62faf38e1e53dfe36"
server_key_version: "encryptionkey@0"
I20260618 22:38:55.310688 9256 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260618 22:38:55.313958 9271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:55.315297 9256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260618 22:38:55.315442 9256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "e2155b2dbd2d4c82aebf8e464b56b5e5"
format_stamp: "Formatted at 2026-06-18 22:38:55 on dist-test-slave-lnj3"
server_key: "893cdd3cd460b95cc8f8bc66d3863178"
server_key_iv: "9a4b016230961cd62faf38e1e53dfe36"
server_key_version: "encryptionkey@0"
I20260618 22:38:55.315608 9256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:55.330097 9256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:55.333834 9256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:55.334113 9256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:55.335125 9256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:38:55.336432 9256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:55.336505 9256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:55.336622 9256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:55.336676 9256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:55.365679 9256 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:36991
I20260618 22:38:55.365712 9383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:36991 every 8 connection(s)
I20260618 22:38:55.366942 9256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:38:55.373718 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 9256
I20260618 22:38:55.373871 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:38:55.374262 5975 external_mini_cluster.cc:1428] Setting key a316f716fe4a9376e2d2964cf9ac1b52
Jun 18 22:38:55 dist-test-slave-lnj3 krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822335, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:55.385918 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:55.369298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:47647 (local address 127.5.213.254:41515)
0618 22:38:55.369455 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:55.369460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:55.370556 (+ 1096us) server_negotiation.cc:407] Connection header received
0618 22:38:55.372210 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:55.372214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:55.372279 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:55.372360 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:55.373747 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:55.374542 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:55.376496 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:55.376908 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:55.377024 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:55.380652 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:55.380658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:55.380663 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:55.380700 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:55.383086 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:55.383897 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:55.383901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:55.383903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:55.384007 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:55.384402 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:55.384405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:55.384407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:55.384635 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:55.384731 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:55.385469 (+ 738us) server_negotiation.cc:299] Negotiation successful
0618 22:38:55.385632 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:38:55.387446 9386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:55.369721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:41515 (local address 127.5.213.194:47647)
0618 22:38:55.370397 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:55.370439 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:55.371976 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:55.372548 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:55.372555 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:55.373009 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:55.373582 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:55.373601 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:55.374714 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:55.374719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:55.376367 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:55.376380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:55.377135 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:55.377141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:55.377442 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:55.378092 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:55.378120 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:55.380454 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:55.383209 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:55.383218 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:55.383237 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:55.383802 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:55.384090 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:55.384094 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:55.384097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:55.384316 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:55.384738 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:55.384746 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:55.385152 (+ 406us) client_negotiation.cc:769] Sending connection context
0618 22:38:55.385420 (+ 268us) client_negotiation.cc:240] Negotiation successful
0618 22:38:55.386037 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":252,"threads_started":1}
I20260618 22:38:55.388965 9384 heartbeater.cc:347] Connected to a master server at 127.5.213.254:41515
I20260618 22:38:55.389281 9384 heartbeater.cc:464] Registering TS with master...
I20260618 22:38:55.390123 9384 heartbeater.cc:511] Master 127.5.213.254:41515 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.213.195@KRBTEST.COM; defaulting to no policy
I20260618 22:38:55.392544 9062 ts_manager.cc:194] Registered new tserver with Master: e2155b2dbd2d4c82aebf8e464b56b5e5 (127.5.213.194:36991)
I20260618 22:38:55.393306 9062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:47647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
I20260618 22:38:55.413142 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:41515
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:55.722420 9389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:55.722743 9389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:55.722853 9389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:55.729877 9389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:55.729974 9389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:55.730083 9389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:38:55.739650 9389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:41515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:55.740796 9389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:55.742022 9389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:55.753989 9394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:55.754020 9397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:55.754078 9395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:55.754719 9389 server_base.cc:1034] running on GCE node
I20260618 22:38:55.755216 9389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:55.756075 9389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:55.757267 9389 hybrid_clock.cc:648] HybridClock initialized: now 1781822335757207 us; error 54 us; skew 500 ppm
I20260618 22:38:55.757594 9389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:55 dist-test-slave-lnj3 krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822335, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:55.760710 9389 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:38:55.761924 9389 webserver.cc:460] Webserver started at http://127.5.213.195:34175/ using document root <none> and password file <none>
I20260618 22:38:55.762504 9389 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:55.762602 9389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:55.762877 9389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:55.765442 9389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "002231a70e2e4f69a3e4f52fda6d68b0"
format_stamp: "Formatted at 2026-06-18 22:38:55 on dist-test-slave-lnj3"
server_key: "392066b72aadaad61ddad613e06bdc3a"
server_key_iv: "b3dfc3347799527001c8c3c7cf31d34c"
server_key_version: "encryptionkey@0"
I20260618 22:38:55.766109 9389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "002231a70e2e4f69a3e4f52fda6d68b0"
format_stamp: "Formatted at 2026-06-18 22:38:55 on dist-test-slave-lnj3"
server_key: "392066b72aadaad61ddad613e06bdc3a"
server_key_iv: "b3dfc3347799527001c8c3c7cf31d34c"
server_key_version: "encryptionkey@0"
I20260618 22:38:55.770766 9389 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20260618 22:38:55.773758 9404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:55.774991 9389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:38:55.775106 9389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "002231a70e2e4f69a3e4f52fda6d68b0"
format_stamp: "Formatted at 2026-06-18 22:38:55 on dist-test-slave-lnj3"
server_key: "392066b72aadaad61ddad613e06bdc3a"
server_key_iv: "b3dfc3347799527001c8c3c7cf31d34c"
server_key_version: "encryptionkey@0"
I20260618 22:38:55.775243 9389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:55.815938 9389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:55.820173 9389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:55.820415 9389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:55.821278 9389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:38:55.822510 9389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:55.822588 9389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:55.822705 9389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:55.822752 9389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:55.850984 9389 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:36859
I20260618 22:38:55.851030 9516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:36859 every 8 connection(s)
I20260618 22:38:55.852125 9389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:38:55.859102 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 9389
I20260618 22:38:55.859282 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:38:55.859826 5975 external_mini_cluster.cc:1428] Setting key 130a4c9d008780fc37f0fc39ca41f610
Jun 18 22:38:55 dist-test-slave-lnj3 krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822335, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:55.870483 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:55.854371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:38379 (local address 127.5.213.254:41515)
0618 22:38:55.854533 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:55.854537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:55.855650 (+ 1113us) server_negotiation.cc:407] Connection header received
0618 22:38:55.857356 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:55.857362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:55.857435 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:55.857516 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:55.859048 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:55.860030 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:55.861733 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:55.862138 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:55.862247 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:55.865365 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:55.865370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:55.865374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:55.865405 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:55.867612 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:55.868264 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:55.868268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:55.868270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:55.868391 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:55.868846 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:55.868852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:55.868855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:55.869138 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:55.869281 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:55.869958 (+ 677us) server_negotiation.cc:299] Negotiation successful
0618 22:38:55.870185 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:38:55.871412 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:55.854864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:41515 (local address 127.5.213.195:38379)
0618 22:38:55.855502 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:55.855542 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:55.857101 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:55.857740 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:55.857748 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:55.858217 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:55.858853 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:55.858878 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:55.860179 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:55.860184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:55.861610 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:55.861622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:55.862339 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:55.862346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:55.862554 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:55.863167 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:55.863191 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:55.865170 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:55.867733 (+ 2563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:55.867742 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:55.867758 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:55.868154 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:55.868477 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:55.868482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:55.868486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:55.868652 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:55.869280 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:55.869288 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:55.869594 (+ 306us) client_negotiation.cc:769] Sending connection context
0618 22:38:55.869841 (+ 247us) client_negotiation.cc:240] Negotiation successful
0618 22:38:55.870340 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":254,"threads_started":1}
I20260618 22:38:55.872865 9517 heartbeater.cc:347] Connected to a master server at 127.5.213.254:41515
I20260618 22:38:55.873153 9517 heartbeater.cc:464] Registering TS with master...
I20260618 22:38:55.873929 9517 heartbeater.cc:511] Master 127.5.213.254:41515 requested a full tablet report, sending...
I20260618 22:38:55.875283 9062 ts_manager.cc:194] Registered new tserver with Master: 002231a70e2e4f69a3e4f52fda6d68b0 (127.5.213.195:36859)
I20260618 22:38:55.876011 9062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:38379
I20260618 22:38:55.886272 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 22:38:55 dist-test-slave-lnj3 krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822335, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 18 22:38:55 dist-test-slave-lnj3 krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822335, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:55.932801 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:55.916097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44544 (local address 127.5.213.254:41515)
0618 22:38:55.916341 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:55.916349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:55.916711 (+ 362us) server_negotiation.cc:407] Connection header received
0618 22:38:55.916865 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:55.916868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:55.916933 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:55.917014 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:55.918620 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:55.919379 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:55.920629 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:55.921057 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:55.921149 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:55.924456 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:55.924461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:55.924465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:55.924496 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:55.930534 (+ 6038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:55.931214 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:55.931218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:55.931221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:55.931351 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:55.931630 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:55.931633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:55.931635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:55.931929 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:55.932103 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:55.932306 (+ 203us) server_negotiation.cc:299] Negotiation successful
0618 22:38:55.932471 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260618 22:38:55.933393 9062 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44544
I20260618 22:38:55.935272 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 9123
I20260618 22:38:55.942728 9251 heartbeater.cc:503] Master 127.5.213.254:41515 was elected leader, sending a full tablet report...
Jun 18 22:38:55 dist-test-slave-lnj3 krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822335, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:38:55.949831 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:55.936262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34544 (local address 127.5.213.193:36047)
0618 22:38:55.936860 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:55.936867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:55.936889 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:38:55.936939 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:55.936944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:55.937141 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:55.937273 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:55.938368 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:55.939408 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:55.940583 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:55.941135 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:55.941245 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:55.945112 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:55.945121 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:55.945136 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:55.945170 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:55.947622 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:55.948139 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:55.948145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:55.948148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:55.948257 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:55.948599 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:55.948601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:55.948603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:55.948921 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:55.949126 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:55.949350 (+ 224us) server_negotiation.cc:299] Negotiation successful
0618 22:38:55.949528 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":275,"threads_started":1}
W20260618 22:38:55.950995 9246 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34544
W20260618 22:38:55.951972 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 9123: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:38:55.952139 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 9123
I20260618 22:38:55.963955 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 9256
Jun 18 22:38:55 dist-test-slave-lnj3 krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822335, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.213.194@KRBTEST.COM
I20260618 22:38:55.977639 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:55.964669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51254 (local address 127.5.213.194:36991)
0618 22:38:55.965253 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:55.965268 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:55.965294 (+ 26us) server_negotiation.cc:407] Connection header received
0618 22:38:55.965354 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:55.965360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:55.965566 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:55.965692 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:55.966748 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:55.967753 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:55.968887 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:55.969421 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:55.969510 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:55.973084 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:55.973096 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:55.973110 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:55.973140 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:55.975571 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:55.976082 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:55.976088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:55.976090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:55.976199 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:55.976453 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:55.976456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:55.976458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:55.976763 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:55.976972 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:55.977168 (+ 196us) server_negotiation.cc:299] Negotiation successful
0618 22:38:55.977362 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":321,"threads_started":1}
W20260618 22:38:55.978979 9379 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51254
W20260618 22:38:55.980238 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 9256: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:38:55.980382 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 9256
I20260618 22:38:55.995563 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 9389
Jun 18 22:38:56 dist-test-slave-lnj3 krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822335, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.213.195@KRBTEST.COM
I20260618 22:38:56.010043 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:55.996458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42962 (local address 127.5.213.195:36859)
0618 22:38:55.996976 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:55.996983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:55.997010 (+ 27us) server_negotiation.cc:407] Connection header received
0618 22:38:55.997063 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:55.997069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:55.997285 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:55.997401 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:55.998554 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:55.999583 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:56.000744 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:56.001302 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:56.001397 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:56.005289 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:56.005300 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:56.005315 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:56.005352 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:56.007845 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:56.008409 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:56.008416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:56.008418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:56.008527 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:56.008786 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:56.008789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:56.008791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:56.009105 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:56.009333 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:56.009529 (+ 196us) server_negotiation.cc:299] Negotiation successful
0618 22:38:56.009716 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":245,"threads_started":1}
W20260618 22:38:56.011303 9512 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42962
W20260618 22:38:56.012305 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 9389: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:38:56.012461 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 9389
I20260618 22:38:56.024518 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 9030
I20260618 22:38:56.025597 9092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:38:56.139616 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 9030
2026-06-18T22:38:56Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3314 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:38:56 dist-test-slave-lnj3 krb5kdc[9534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:38:56 dist-test-slave-lnj3 krb5kdc[9534](info): set up 2 sockets
Jun 18 22:38:56 dist-test-slave-lnj3 krb5kdc[9534](info): commencing operation
krb5kdc: starting...
W20260618 22:38:57.130765 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:38:57 dist-test-slave-lnj3 krb5kdc[9534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:38:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:38:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:38:57.306928 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:36359
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:36359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:57.617574 9547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:57.617863 9547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:57.617974 9547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:57.625151 9547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:38:57.625257 9547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:57.625311 9547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:57.625385 9547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:38:57.625434 9547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:38:57.635182 9547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:36359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:36359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:57.636735 9547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:57.638134 9547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:57.649652 9552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:57.649609 9555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:57.649585 9553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:57.650553 9547 server_base.cc:1034] running on GCE node
I20260618 22:38:57.651400 9547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:57.652848 9547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:57.654062 9547 hybrid_clock.cc:648] HybridClock initialized: now 1781822337654009 us; error 46 us; skew 500 ppm
I20260618 22:38:57.654390 9547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:57 dist-test-slave-lnj3 krb5kdc[9534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822337, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:57.662710 9547 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:38:57.664288 9547 webserver.cc:460] Webserver started at http://127.5.213.254:43401/ using document root <none> and password file <none>
I20260618 22:38:57.664991 9547 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:57.665109 9547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:57.665407 9547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:57.668397 9547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "fef1a3ea40a5498b91254439d6fda58c"
format_stamp: "Formatted at 2026-06-18 22:38:57 on dist-test-slave-lnj3"
server_key: "cc7f59307b1fb60be9c4d1ded24cccda"
server_key_iv: "06bd23729121a55e1092ace717e2ec4c"
server_key_version: "encryptionkey@0"
I20260618 22:38:57.669167 9547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "fef1a3ea40a5498b91254439d6fda58c"
format_stamp: "Formatted at 2026-06-18 22:38:57 on dist-test-slave-lnj3"
server_key: "cc7f59307b1fb60be9c4d1ded24cccda"
server_key_iv: "06bd23729121a55e1092ace717e2ec4c"
server_key_version: "encryptionkey@0"
I20260618 22:38:57.674397 9547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260618 22:38:57.677891 9562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:57.679348 9547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:38:57.679504 9547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "fef1a3ea40a5498b91254439d6fda58c"
format_stamp: "Formatted at 2026-06-18 22:38:57 on dist-test-slave-lnj3"
server_key: "cc7f59307b1fb60be9c4d1ded24cccda"
server_key_iv: "06bd23729121a55e1092ace717e2ec4c"
server_key_version: "encryptionkey@0"
I20260618 22:38:57.679683 9547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:57.722767 9547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:57.727125 9547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:57.727398 9547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:57.744149 9547 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:36359
I20260618 22:38:57.744192 9613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:36359 every 8 connection(s)
I20260618 22:38:57.745476 9547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:38:57.749176 9614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:38:57.754110 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 9547
I20260618 22:38:57.754299 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:38:57.754930 5975 external_mini_cluster.cc:1428] Setting key e655731a51359c21c3eefbf4f866e6f0
I20260618 22:38:57.760186 9614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c: Bootstrap starting.
I20260618 22:38:57.763725 9614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c: Neither blocks nor log segments found. Creating new log.
I20260618 22:38:57.764842 9614 log.cc:824] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c: Log is configured to *not* fsync() on all Append() calls
I20260618 22:38:57.767727 9614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c: No bootstrap required, opened a new log
Jun 18 22:38:57 dist-test-slave-lnj3 krb5kdc[9534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:57.776930 9614 raft_consensus.cc:348] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef1a3ea40a5498b91254439d6fda58c" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36359 } }
I20260618 22:38:57.777374 9614 raft_consensus.cc:374] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:38:57.777478 9614 raft_consensus.cc:729] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef1a3ea40a5498b91254439d6fda58c, State: Initialized, Role: FOLLOWER
I20260618 22:38:57.778137 9614 consensus_queue.cc:260] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef1a3ea40a5498b91254439d6fda58c" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36359 } }
I20260618 22:38:57.778409 9614 raft_consensus.cc:388] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:38:57.778543 9614 raft_consensus.cc:482] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:38:57.778825 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:57.756986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51766 (local address 127.5.213.254:36359)
0618 22:38:57.757765 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:57.757782 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:57.757835 (+ 53us) server_negotiation.cc:407] Connection header received
0618 22:38:57.760102 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:57.760131 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:57.760628 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:57.761050 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:57.762306 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:57.764239 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:57.765664 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:57.766517 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:57.766714 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:57.770328 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:57.770342 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:57.770362 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:57.770407 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:57.774268 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:57.774986 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:57.774995 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:57.775004 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:57.775221 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:57.775534 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:57.775539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:57.775543 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:57.776245 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:57.776540 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:57.776977 (+ 437us) server_negotiation.cc:299] Negotiation successful
0618 22:38:57.777430 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":314,"threads_started":1}
I20260618 22:38:57.778817 9614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:38:57.781677 9614 raft_consensus.cc:504] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef1a3ea40a5498b91254439d6fda58c" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36359 } }
I20260618 22:38:57.782147 9614 leader_election.cc:302] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fef1a3ea40a5498b91254439d6fda58c; no voters:
I20260618 22:38:57.782692 9614 leader_election.cc:288] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:38:57.782824 9619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:38:57.783892 9619 raft_consensus.cc:686] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c [term 1 LEADER]: Becoming Leader. State: Replica: fef1a3ea40a5498b91254439d6fda58c, State: Running, Role: LEADER
I20260618 22:38:57.784443 9619 consensus_queue.cc:237] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef1a3ea40a5498b91254439d6fda58c" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36359 } }
I20260618 22:38:57.784850 9614 sys_catalog.cc:564] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:38:57.787357 9620 sys_catalog.cc:455] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fef1a3ea40a5498b91254439d6fda58c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef1a3ea40a5498b91254439d6fda58c" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36359 } } }
I20260618 22:38:57.787770 9620 sys_catalog.cc:458] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c [sys.catalog]: This master's current role is: LEADER
I20260618 22:38:57.787348 9621 sys_catalog.cc:455] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fef1a3ea40a5498b91254439d6fda58c. Latest consensus state: current_term: 1 leader_uuid: "fef1a3ea40a5498b91254439d6fda58c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef1a3ea40a5498b91254439d6fda58c" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36359 } } }
I20260618 22:38:57.788229 9621 sys_catalog.cc:458] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c [sys.catalog]: This master's current role is: LEADER
I20260618 22:38:57.790055 9626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:38:57.795118 9626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:38:57.805105 9626 catalog_manager.cc:1347] Generated new cluster ID: 44fb06e1c3584a86a01e087fce4e7750
I20260618 22:38:57.805230 9626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:38:57.838049 9626 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:38:57.839406 9626 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:38:57.850841 9626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fef1a3ea40a5498b91254439d6fda58c: Generated new TSK 0
I20260618 22:38:57.851850 9626 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
I20260618 22:38:57.902488 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36359
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:58.217208 9640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:58.217469 9640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:58.217586 9640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:58.224680 9640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:58.224792 9640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:58.224937 9640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:38:58.234423 9640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:58.235641 9640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:58.236910 9640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:58.248908 9645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:58.248915 9646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:58.248946 9648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:58.249807 9640 server_base.cc:1034] running on GCE node
I20260618 22:38:58.250377 9640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:58.251312 9640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:58.252501 9640 hybrid_clock.cc:648] HybridClock initialized: now 1781822338252430 us; error 68 us; skew 500 ppm
I20260618 22:38:58.252807 9640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:58 dist-test-slave-lnj3 krb5kdc[9534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822338, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:58.255694 9640 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:38:58.257159 9640 webserver.cc:460] Webserver started at http://127.5.213.193:45717/ using document root <none> and password file <none>
I20260618 22:38:58.257802 9640 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:58.257916 9640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:58.258196 9640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:58.261044 9640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "f441f9436a32485b809d2a239d36bb81"
format_stamp: "Formatted at 2026-06-18 22:38:58 on dist-test-slave-lnj3"
server_key: "366c0a9a8b6d5643002067c0a79891dd"
server_key_iv: "003faf5a5a95a66ab649eed3307c45f2"
server_key_version: "encryptionkey@0"
I20260618 22:38:58.261790 9640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "f441f9436a32485b809d2a239d36bb81"
format_stamp: "Formatted at 2026-06-18 22:38:58 on dist-test-slave-lnj3"
server_key: "366c0a9a8b6d5643002067c0a79891dd"
server_key_iv: "003faf5a5a95a66ab649eed3307c45f2"
server_key_version: "encryptionkey@0"
I20260618 22:38:58.266803 9640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260618 22:38:58.269984 9655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:58.271268 9640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:38:58.271402 9640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "f441f9436a32485b809d2a239d36bb81"
format_stamp: "Formatted at 2026-06-18 22:38:58 on dist-test-slave-lnj3"
server_key: "366c0a9a8b6d5643002067c0a79891dd"
server_key_iv: "003faf5a5a95a66ab649eed3307c45f2"
server_key_version: "encryptionkey@0"
I20260618 22:38:58.271569 9640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:58.286603 9640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:58.290227 9640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:58.290475 9640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:58.291355 9640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:38:58.292680 9640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:58.292754 9640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:58.292865 9640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:58.292917 9640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:58.322237 9640 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:35971
I20260618 22:38:58.322276 9767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:35971 every 8 connection(s)
I20260618 22:38:58.323494 9640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:38:58.330484 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 9640
I20260618 22:38:58.330742 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:38:58.331434 5975 external_mini_cluster.cc:1428] Setting key 1c4620b0a1477c692a0a4dea8db2bbf7
Jun 18 22:38:58 dist-test-slave-lnj3 krb5kdc[9534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822338, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:58.343994 9770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:58.325860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:49309 (local address 127.5.213.254:36359)
0618 22:38:58.326300 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:58.326306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:58.327221 (+ 915us) server_negotiation.cc:407] Connection header received
0618 22:38:58.328982 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:58.328987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:58.329165 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:58.329276 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:58.331108 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:58.331865 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:58.333733 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:58.334124 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:58.334217 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:58.338015 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:58.338020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:58.338024 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:58.338055 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:58.341023 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:58.341732 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:58.341737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:58.341741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:58.341870 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:58.342316 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:58.342320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:58.342323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:58.342669 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:58.342802 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:58.343459 (+ 657us) server_negotiation.cc:299] Negotiation successful
0618 22:38:58.343657 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":202,"threads_started":1}
I20260618 22:38:58.344817 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:58.326371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:36359 (local address 127.5.213.193:49309)
0618 22:38:58.327071 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:58.327115 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:58.328733 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:58.329524 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:58.329534 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:58.330052 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:58.330928 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:58.330957 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:58.332014 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:58.332018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:58.333606 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:58.333619 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:58.334332 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:58.334339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:58.334679 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:58.335340 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:58.335365 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:58.337808 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:58.341164 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:58.341172 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:58.341185 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:58.341596 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:58.341991 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:58.341995 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:58.341997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:58.342208 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:58.342800 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:58.342806 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:58.343128 (+ 322us) client_negotiation.cc:769] Sending connection context
0618 22:38:58.343352 (+ 224us) client_negotiation.cc:240] Negotiation successful
0618 22:38:58.343721 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":270,"threads_started":1}
I20260618 22:38:58.346318 9768 heartbeater.cc:347] Connected to a master server at 127.5.213.254:36359
I20260618 22:38:58.346678 9768 heartbeater.cc:464] Registering TS with master...
I20260618 22:38:58.347563 9768 heartbeater.cc:511] Master 127.5.213.254:36359 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.213.194@KRBTEST.COM; defaulting to no policy
I20260618 22:38:58.349900 9579 ts_manager.cc:194] Registered new tserver with Master: f441f9436a32485b809d2a239d36bb81 (127.5.213.193:35971)
I20260618 22:38:58.351738 9579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:49309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
I20260618 22:38:58.371371 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36359
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:58.677500 9774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:58.677800 9774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:58.677933 9774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:58.685065 9774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:58.685194 9774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:58.685348 9774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:38:58.695005 9774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:58.696250 9774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:58.697574 9774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:58.709430 9780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:58.709611 9782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:58.709645 9779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:58.710541 9774 server_base.cc:1034] running on GCE node
I20260618 22:38:58.711153 9774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:58.711997 9774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:58.713213 9774 hybrid_clock.cc:648] HybridClock initialized: now 1781822338713168 us; error 32 us; skew 500 ppm
I20260618 22:38:58.713507 9774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:58 dist-test-slave-lnj3 krb5kdc[9534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822338, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:58.716467 9774 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:38:58.717830 9774 webserver.cc:460] Webserver started at http://127.5.213.194:43715/ using document root <none> and password file <none>
I20260618 22:38:58.718466 9774 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:58.718582 9774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:58.718924 9774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:58.721659 9774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "252c78e72ba54cb18c048c7ea86ec220"
format_stamp: "Formatted at 2026-06-18 22:38:58 on dist-test-slave-lnj3"
server_key: "a5f44bc1ffe9121b2c2b35671d5294f6"
server_key_iv: "2db5d0c31ed3d9f0286844347059b79f"
server_key_version: "encryptionkey@0"
I20260618 22:38:58.722368 9774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "252c78e72ba54cb18c048c7ea86ec220"
format_stamp: "Formatted at 2026-06-18 22:38:58 on dist-test-slave-lnj3"
server_key: "a5f44bc1ffe9121b2c2b35671d5294f6"
server_key_iv: "2db5d0c31ed3d9f0286844347059b79f"
server_key_version: "encryptionkey@0"
I20260618 22:38:58.727262 9774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260618 22:38:58.730427 9789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:58.731642 9774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:38:58.731812 9774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "252c78e72ba54cb18c048c7ea86ec220"
format_stamp: "Formatted at 2026-06-18 22:38:58 on dist-test-slave-lnj3"
server_key: "a5f44bc1ffe9121b2c2b35671d5294f6"
server_key_iv: "2db5d0c31ed3d9f0286844347059b79f"
server_key_version: "encryptionkey@0"
I20260618 22:38:58.731988 9774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:58.753971 9774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:58.758844 9774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:58.759130 9774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:58.760183 9774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:38:58.761677 9774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:58.761759 9774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:58.761865 9774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:58.761916 9774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:58.790764 9774 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:34771
I20260618 22:38:58.790817 9901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:34771 every 8 connection(s)
I20260618 22:38:58.791939 9774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:38:58.796687 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 9774
I20260618 22:38:58.796866 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:38:58.797365 5975 external_mini_cluster.cc:1428] Setting key 8fde61ebd5c3383106011f4d3778bedc
Jun 18 22:38:58 dist-test-slave-lnj3 krb5kdc[9534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822338, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:58.812964 9770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:58.794338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:55139 (local address 127.5.213.254:36359)
0618 22:38:58.794518 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:58.794524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:58.795890 (+ 1366us) server_negotiation.cc:407] Connection header received
0618 22:38:58.798100 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:58.798106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:58.798178 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:58.798269 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:58.800078 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:58.800795 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:58.802946 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:58.803422 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:58.803517 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:58.807113 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:58.807120 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:58.807125 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:58.807162 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:58.809993 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:58.810691 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:58.810697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:58.810700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:58.810859 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:58.811234 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:58.811238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:58.811242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:58.811562 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:58.811720 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:58.812388 (+ 668us) server_negotiation.cc:299] Negotiation successful
0618 22:38:58.812615 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:38:58.813691 9904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:58.795014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:36359 (local address 127.5.213.194:55139)
0618 22:38:58.795726 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:58.795765 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:58.797821 (+ 2056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:58.798509 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:58.798517 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:58.799121 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:58.799905 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:58.799930 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:58.800938 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:58.800942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:58.802815 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:58.802829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:58.803630 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:58.803637 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:58.803920 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:58.804604 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:58.804631 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:58.806875 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:58.810124 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:58.810131 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:58.810144 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:58.810552 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:58.810953 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:58.810956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:58.810958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:58.811125 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:58.811676 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:58.811684 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:58.812008 (+ 324us) client_negotiation.cc:769] Sending connection context
0618 22:38:58.812235 (+ 227us) client_negotiation.cc:240] Negotiation successful
0618 22:38:58.812614 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":253,"threads_started":1}
I20260618 22:38:58.815022 9902 heartbeater.cc:347] Connected to a master server at 127.5.213.254:36359
I20260618 22:38:58.815346 9902 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.213.195@KRBTEST.COM; defaulting to no policy
I20260618 22:38:58.816135 9902 heartbeater.cc:511] Master 127.5.213.254:36359 requested a full tablet report, sending...
I20260618 22:38:58.817515 9579 ts_manager.cc:194] Registered new tserver with Master: 252c78e72ba54cb18c048c7ea86ec220 (127.5.213.194:34771)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.195@KRBTEST.COM" created.
I20260618 22:38:58.818509 9579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:55139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
I20260618 22:38:58.839413 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36359
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:38:59.145399 9907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:38:59.145699 9907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:38:59.145828 9907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:38:59.153029 9907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:38:59.153163 9907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:38:59.153319 9907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:38:59.163084 9907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:38:59.164373 9907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:38:59.165699 9907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:38:59.178110 9912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:59.178136 9913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:38:59.178364 9915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:38:59.179050 9907 server_base.cc:1034] running on GCE node
I20260618 22:38:59.179603 9907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:38:59.180509 9907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:38:59.181700 9907 hybrid_clock.cc:648] HybridClock initialized: now 1781822339181652 us; error 41 us; skew 500 ppm
I20260618 22:38:59.182011 9907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:38:59 dist-test-slave-lnj3 krb5kdc[9534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822339, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:38:59.185185 9907 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:38:59.186563 9907 webserver.cc:460] Webserver started at http://127.5.213.195:35009/ using document root <none> and password file <none>
I20260618 22:38:59.187296 9907 fs_manager.cc:359] Metadata directory not provided
I20260618 22:38:59.187402 9907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:38:59.187700 9907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:38:59.190552 9907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "c66e875a67b84a2b88a9ebbc39a10ddb"
format_stamp: "Formatted at 2026-06-18 22:38:59 on dist-test-slave-lnj3"
server_key: "98f5d937e479e579c8e1365f54d54303"
server_key_iv: "d7a6c792c355a9a8906fc72be8c10af1"
server_key_version: "encryptionkey@0"
I20260618 22:38:59.191358 9907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "c66e875a67b84a2b88a9ebbc39a10ddb"
format_stamp: "Formatted at 2026-06-18 22:38:59 on dist-test-slave-lnj3"
server_key: "98f5d937e479e579c8e1365f54d54303"
server_key_iv: "d7a6c792c355a9a8906fc72be8c10af1"
server_key_version: "encryptionkey@0"
I20260618 22:38:59.196650 9907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:38:59.200155 9922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:59.201568 9907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:38:59.201763 9907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "c66e875a67b84a2b88a9ebbc39a10ddb"
format_stamp: "Formatted at 2026-06-18 22:38:59 on dist-test-slave-lnj3"
server_key: "98f5d937e479e579c8e1365f54d54303"
server_key_iv: "d7a6c792c355a9a8906fc72be8c10af1"
server_key_version: "encryptionkey@0"
I20260618 22:38:59.201946 9907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:38:59.216463 9907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:38:59.220512 9907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:38:59.220777 9907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:38:59.221673 9907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:38:59.222937 9907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:38:59.223007 9907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:59.223101 9907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:38:59.223156 9907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:38:59.251659 9907 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:37343
I20260618 22:38:59.251698 10034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:37343 every 8 connection(s)
I20260618 22:38:59.252934 9907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:38:59.255514 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 9907
I20260618 22:38:59.255686 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:38:59.256104 5975 external_mini_cluster.cc:1428] Setting key b2dff31dce53cf53e2cb1c757eff6929
Jun 18 22:38:59 dist-test-slave-lnj3 krb5kdc[9534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822339, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:59.271044 9770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:59.255297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:33261 (local address 127.5.213.254:36359)
0618 22:38:59.255488 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:59.255495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:59.256789 (+ 1294us) server_negotiation.cc:407] Connection header received
0618 22:38:59.258527 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:59.258532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:59.258611 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:59.258723 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:59.260119 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:59.260821 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:59.262272 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:59.262703 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:59.262806 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:59.265894 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:59.265899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:59.265902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:59.265932 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:59.268240 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:59.268903 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:59.268908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:59.268910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:59.269019 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:59.269398 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:59.269402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:59.269405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:59.269668 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:59.269798 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:59.270444 (+ 646us) server_negotiation.cc:299] Negotiation successful
0618 22:38:59.270695 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 22:38:59.271937 10037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:59.255841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:36359 (local address 127.5.213.195:33261)
0618 22:38:59.256589 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0618 22:38:59.256683 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0618 22:38:59.258284 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:38:59.258950 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:38:59.258958 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:38:59.259432 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:38:59.259972 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:59.259987 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:59.260955 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:59.260959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:59.262149 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:38:59.262160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:38:59.262891 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:38:59.262898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:38:59.263118 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:59.263707 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:38:59.263727 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:38:59.265704 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:38:59.268380 (+ 2676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:59.268389 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:59.268406 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:59.268799 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:59.269121 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:38:59.269125 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:38:59.269128 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:38:59.269298 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:38:59.269791 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:38:59.269799 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:38:59.270123 (+ 324us) client_negotiation.cc:769] Sending connection context
0618 22:38:59.270352 (+ 229us) client_negotiation.cc:240] Negotiation successful
0618 22:38:59.270890 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":297,"threads_started":1}
I20260618 22:38:59.273104 10035 heartbeater.cc:347] Connected to a master server at 127.5.213.254:36359
I20260618 22:38:59.273388 10035 heartbeater.cc:464] Registering TS with master...
I20260618 22:38:59.274051 10035 heartbeater.cc:511] Master 127.5.213.254:36359 requested a full tablet report, sending...
I20260618 22:38:59.275395 9579 ts_manager.cc:194] Registered new tserver with Master: c66e875a67b84a2b88a9ebbc39a10ddb (127.5.213.195:37343)
I20260618 22:38:59.276175 9579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:33261
I20260618 22:38:59.282913 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 22:38:59 dist-test-slave-lnj3 krb5kdc[9534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 22:38:59 dist-test-slave-lnj3 krb5kdc[9534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:38:59.326676 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:59.312237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55984 (local address 127.5.213.193:35971)
0618 22:38:59.312815 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:59.312822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:59.312851 (+ 29us) server_negotiation.cc:407] Connection header received
0618 22:38:59.312965 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:59.312972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:59.313182 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:59.313317 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:59.314597 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:59.315628 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:59.316910 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:59.317475 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:59.317587 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:59.321137 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:59.321148 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:59.321162 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:59.321193 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:59.324027 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:59.324647 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:59.324654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:59.324657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:59.324771 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:59.325137 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:59.325142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:59.325145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:59.325500 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:59.325774 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:59.326062 (+ 288us) server_negotiation.cc:299] Negotiation successful
0618 22:38:59.326318 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":290,"threads_started":1}
Jun 18 22:38:59 dist-test-slave-lnj3 krb5kdc[9534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:38:59.342669 9770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:59.330365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51780 (local address 127.5.213.254:36359)
0618 22:38:59.330567 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:59.330573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:59.330699 (+ 126us) server_negotiation.cc:407] Connection header received
0618 22:38:59.330836 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:59.330840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:59.330901 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:59.330976 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:59.332056 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:59.332771 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:59.333860 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:59.334226 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:59.334310 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:59.337877 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:59.337882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:59.337886 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:59.337920 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:59.340354 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:59.341003 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:59.341008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:59.341010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:59.341150 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:59.341500 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:59.341503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:59.341506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:59.341808 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:59.341995 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:59.342199 (+ 204us) server_negotiation.cc:299] Negotiation successful
0618 22:38:59.342359 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260618 22:38:59.343211 9579 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51780
I20260618 22:38:59.344084 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 9640
I20260618 22:38:59.355213 9768 heartbeater.cc:503] Master 127.5.213.254:36359 was elected leader, sending a full tablet report...
I20260618 22:38:59.355769 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:59.344885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55992 (local address 127.5.213.193:35971)
0618 22:38:59.345033 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:59.345037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:59.345386 (+ 349us) server_negotiation.cc:407] Connection header received
0618 22:38:59.345503 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:59.345506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:59.345579 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:59.345654 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:59.346776 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:59.347461 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:59.348578 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:59.348990 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:59.349079 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:59.351237 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:59.351242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:59.351245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:59.351282 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:59.353625 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:59.354167 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:59.354170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:59.354172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:59.354285 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:59.354527 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:59.354529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:59.354531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:59.354813 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:59.354999 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:59.355276 (+ 277us) server_negotiation.cc:299] Negotiation successful
0618 22:38:59.355479 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:38:59.356235 9763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:38:59.436069 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 9640
I20260618 22:38:59.448400 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 9774
Jun 18 22:38:59 dist-test-slave-lnj3 krb5kdc[9534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.194@KRBTEST.COM
I20260618 22:38:59.462953 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:59.449225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.5.213.194:34771)
0618 22:38:59.449742 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:59.449750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:59.449775 (+ 25us) server_negotiation.cc:407] Connection header received
0618 22:38:59.449833 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:59.449839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:59.450043 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:59.450190 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:59.451384 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:59.452384 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:59.453603 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:59.454153 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:59.454274 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:59.458164 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:59.458175 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:59.458189 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:59.458229 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:59.460765 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:59.461318 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:59.461325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:59.461327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:59.461443 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:59.461703 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:59.461706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:59.461708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:59.462018 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:59.462243 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:59.462451 (+ 208us) server_negotiation.cc:299] Negotiation successful
0618 22:38:59.462671 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":239,"threads_started":1}
I20260618 22:38:59.464164 9897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:38:59.545509 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 9774
I20260618 22:38:59.557633 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 9907
Jun 18 22:38:59 dist-test-slave-lnj3 krb5kdc[9534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.195@KRBTEST.COM
I20260618 22:38:59.572540 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:38:59.558568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34686 (local address 127.5.213.195:37343)
0618 22:38:59.559174 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0618 22:38:59.559181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:38:59.559209 (+ 28us) server_negotiation.cc:407] Connection header received
0618 22:38:59.559264 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:38:59.559271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:38:59.559486 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:38:59.559645 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:38:59.560739 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:59.561737 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:59.563099 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:38:59.563660 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:38:59.563755 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:38:59.567737 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:38:59.567749 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:38:59.567764 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:38:59.567795 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:38:59.570253 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:59.570866 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:59.570874 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:59.570876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:59.570990 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:38:59.571260 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:38:59.571263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:38:59.571265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:38:59.571566 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:38:59.571814 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0618 22:38:59.572015 (+ 201us) server_negotiation.cc:299] Negotiation successful
0618 22:38:59.572198 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":316,"threads_started":1}
I20260618 22:38:59.573835 10030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:38:59.653707 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 9907
I20260618 22:38:59.665309 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 9547
I20260618 22:38:59.666410 9609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:38:59.764956 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 9547
2026-06-18T22:38:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3625 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:38:59 dist-test-slave-lnj3 krb5kdc[10054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:38:59 dist-test-slave-lnj3 krb5kdc[10054](info): set up 2 sockets
Jun 18 22:38:59 dist-test-slave-lnj3 krb5kdc[10054](info): commencing operation
krb5kdc: starting...
W20260618 22:39:00.758903 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:39:00 dist-test-slave-lnj3 krb5kdc[10054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:39:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:39:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:39:00.932557 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:44807
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:41917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:44807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:01.231856 10067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:01.232117 10067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:01.232249 10067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:01.239342 10067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:39:01.239444 10067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:01.239513 10067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:01.239586 10067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:39:01.239651 10067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:39:01.249349 10067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:41917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:44807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:44807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:01.250522 10067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:01.251734 10067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:01.262313 10075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:01.262344 10073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:01.262499 10072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:01.263096 10067 server_base.cc:1034] running on GCE node
I20260618 22:39:01.263767 10067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:01.265314 10067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:01.266518 10067 hybrid_clock.cc:648] HybridClock initialized: now 1781822341266478 us; error 51 us; skew 500 ppm
I20260618 22:39:01.266870 10067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:01 dist-test-slave-lnj3 krb5kdc[10054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822341, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:01.269735 10067 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:39:01.270772 10067 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:01.270860 10067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:01.271116 10067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:01.273799 10067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "68c949bfd1f64fbc91162efe84c12b7d"
format_stamp: "Formatted at 2026-06-18 22:39:01 on dist-test-slave-lnj3"
server_key: "8dc7466d094a51840ec98a3c5d3a3e29"
server_key_iv: "c9d9468239e63dde931cf7e401195994"
server_key_version: "encryptionkey@0"
I20260618 22:39:01.274489 10067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "68c949bfd1f64fbc91162efe84c12b7d"
format_stamp: "Formatted at 2026-06-18 22:39:01 on dist-test-slave-lnj3"
server_key: "8dc7466d094a51840ec98a3c5d3a3e29"
server_key_iv: "c9d9468239e63dde931cf7e401195994"
server_key_version: "encryptionkey@0"
I20260618 22:39:01.279120 10067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260618 22:39:01.282169 10081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:01.283401 10067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:39:01.283528 10067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "68c949bfd1f64fbc91162efe84c12b7d"
format_stamp: "Formatted at 2026-06-18 22:39:01 on dist-test-slave-lnj3"
server_key: "8dc7466d094a51840ec98a3c5d3a3e29"
server_key_iv: "c9d9468239e63dde931cf7e401195994"
server_key_version: "encryptionkey@0"
I20260618 22:39:01.283694 10067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:01.295011 10067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:01.298846 10067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:01.299053 10067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:01.314452 10067 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:44807
I20260618 22:39:01.314433 10132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:44807 every 8 connection(s)
I20260618 22:39:01.315279 10067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:39:01.316555 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 10067
I20260618 22:39:01.316756 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:39:01.317351 5975 external_mini_cluster.cc:1428] Setting key a7ed6c4723607bae24e3a01677101403
I20260618 22:39:01.319164 10133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 18 22:39:01 dist-test-slave-lnj3 krb5kdc[10054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:39:01.331282 10133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d: Bootstrap starting.
I20260618 22:39:01.335114 10133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:01.336237 10133 log.cc:824] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:01.339262 10133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d: No bootstrap required, opened a new log
I20260618 22:39:01.339510 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:01.319125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45114 (local address 127.5.213.254:44807)
0618 22:39:01.319912 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:01.319927 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:01.319974 (+ 47us) server_negotiation.cc:407] Connection header received
0618 22:39:01.321938 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:01.321965 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:01.322442 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:01.322880 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:01.324049 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:01.326000 (+ 1951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:01.327209 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:01.328161 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:01.328346 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:01.331793 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:01.331803 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:01.331821 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:01.331860 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:01.335363 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:01.335991 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:01.335999 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:01.336006 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:01.336205 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:01.336570 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:01.336575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:01.336578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:01.337182 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:01.337395 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:01.337841 (+ 446us) server_negotiation.cc:299] Negotiation successful
0618 22:39:01.338234 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":334,"threads_started":1}
I20260618 22:39:01.346311 10133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c949bfd1f64fbc91162efe84c12b7d" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44807 } }
I20260618 22:39:01.346650 10133 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:01.346732 10133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68c949bfd1f64fbc91162efe84c12b7d, State: Initialized, Role: FOLLOWER
I20260618 22:39:01.347203 10133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c949bfd1f64fbc91162efe84c12b7d" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44807 } }
I20260618 22:39:01.347426 10133 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:39:01.347554 10133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:39:01.347723 10133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:01.349850 10133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c949bfd1f64fbc91162efe84c12b7d" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44807 } }
I20260618 22:39:01.350368 10133 leader_election.cc:302] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68c949bfd1f64fbc91162efe84c12b7d; no voters:
I20260618 22:39:01.350920 10133 leader_election.cc:288] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:39:01.351017 10138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:01.351712 10138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d [term 1 LEADER]: Becoming Leader. State: Replica: 68c949bfd1f64fbc91162efe84c12b7d, State: Running, Role: LEADER
I20260618 22:39:01.352209 10138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c949bfd1f64fbc91162efe84c12b7d" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44807 } }
I20260618 22:39:01.352756 10133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:39:01.354676 10140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68c949bfd1f64fbc91162efe84c12b7d. Latest consensus state: current_term: 1 leader_uuid: "68c949bfd1f64fbc91162efe84c12b7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c949bfd1f64fbc91162efe84c12b7d" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44807 } } }
I20260618 22:39:01.354995 10140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:01.355666 10139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68c949bfd1f64fbc91162efe84c12b7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c949bfd1f64fbc91162efe84c12b7d" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44807 } } }
I20260618 22:39:01.355957 10139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:01.356613 10143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:39:01.361721 10143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:39:01.369463 10143 catalog_manager.cc:1347] Generated new cluster ID: 6d4a04978fe74b7da81a7acd53896335
I20260618 22:39:01.369566 10143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:39:01.394984 10143 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:39:01.396255 10143 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:39:01.401062 10143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d: Generated new TSK 0
I20260618 22:39:01.402022 10143 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
I20260618 22:39:01.445797 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44807
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:41917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:01.762353 10159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:01.762655 10159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:01.762791 10159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:01.769804 10159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:01.769922 10159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:01.770077 10159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:39:01.779731 10159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:41917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:01.780977 10159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:01.782330 10159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:01.794348 10164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:01.794324 10167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:01.794289 10165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:01.795029 10159 server_base.cc:1034] running on GCE node
I20260618 22:39:01.795607 10159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:01.796587 10159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:01.797837 10159 hybrid_clock.cc:648] HybridClock initialized: now 1781822341797773 us; error 47 us; skew 500 ppm
I20260618 22:39:01.798182 10159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:01 dist-test-slave-lnj3 krb5kdc[10054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822341, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:01.801292 10159 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:39:01.802457 10159 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:01.802557 10159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:01.802911 10159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:01.805689 10159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "f2b384901eaf487f96541c493f34ccd9"
format_stamp: "Formatted at 2026-06-18 22:39:01 on dist-test-slave-lnj3"
server_key: "d8d194cf7f96463216d7f31f3f11545c"
server_key_iv: "60d297c6e9baa5842350c64cc303b4e1"
server_key_version: "encryptionkey@0"
I20260618 22:39:01.806448 10159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "f2b384901eaf487f96541c493f34ccd9"
format_stamp: "Formatted at 2026-06-18 22:39:01 on dist-test-slave-lnj3"
server_key: "d8d194cf7f96463216d7f31f3f11545c"
server_key_iv: "60d297c6e9baa5842350c64cc303b4e1"
server_key_version: "encryptionkey@0"
I20260618 22:39:01.811084 10159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260618 22:39:01.814263 10173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:01.815490 10159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:39:01.815649 10159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "f2b384901eaf487f96541c493f34ccd9"
format_stamp: "Formatted at 2026-06-18 22:39:01 on dist-test-slave-lnj3"
server_key: "d8d194cf7f96463216d7f31f3f11545c"
server_key_iv: "60d297c6e9baa5842350c64cc303b4e1"
server_key_version: "encryptionkey@0"
I20260618 22:39:01.815828 10159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:01.838006 10159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:01.841727 10159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:01.841965 10159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:01.842813 10159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:01.844023 10159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:01.844120 10159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:01.844213 10159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:01.844277 10159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:01.872107 10159 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:39547
I20260618 22:39:01.872156 10285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:39547 every 8 connection(s)
I20260618 22:39:01.872889 10159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:39:01.874099 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 10159
I20260618 22:39:01.874286 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:39:01.874910 5975 external_mini_cluster.cc:1428] Setting key f2fbbee555bc6c183cfdd935153b7e76
Jun 18 22:39:01 dist-test-slave-lnj3 krb5kdc[10054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822341, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.213.194@KRBTEST.COM; defaulting to no policy
I20260618 22:39:01.894098 10288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:01.875199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:52583 (local address 127.5.213.254:44807)
0618 22:39:01.875776 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:01.875781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:01.876643 (+ 862us) server_negotiation.cc:407] Connection header received
0618 22:39:01.879008 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:01.879012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:01.879071 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:01.879155 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:01.881008 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:01.881678 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:01.883520 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:01.884056 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:01.884152 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:01.888030 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:01.888035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:01.888038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:01.888067 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:01.890939 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:01.891649 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:01.891655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:01.891660 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:01.891814 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:01.892190 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:01.892196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:01.892199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:01.892596 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:01.892760 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:01.893412 (+ 652us) server_negotiation.cc:299] Negotiation successful
0618 22:39:01.893658 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":366,"threads_started":1}
I20260618 22:39:01.894816 10289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:01.875729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:44807 (local address 127.5.213.193:52583)
0618 22:39:01.876496 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:01.876552 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:01.878722 (+ 2170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:01.879404 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:01.879416 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:01.880104 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:01.880863 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:01.880886 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:01.881802 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:01.881807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:01.883389 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:01.883400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:01.884302 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:01.884310 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:01.884604 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:01.885283 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:01.885315 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:01.887807 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:01.891090 (+ 3283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:01.891098 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:01.891110 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:01.891518 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:01.891917 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:01.891920 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:01.891922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:01.892081 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:01.892713 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:01.892718 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:01.893020 (+ 302us) client_negotiation.cc:769] Sending connection context
0618 22:39:01.893242 (+ 222us) client_negotiation.cc:240] Negotiation successful
0618 22:39:01.893643 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":275,"threads_started":1}
I20260618 22:39:01.896174 10286 heartbeater.cc:347] Connected to a master server at 127.5.213.254:44807
I20260618 22:39:01.896437 10286 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.194@KRBTEST.COM" created.
I20260618 22:39:01.897035 10286 heartbeater.cc:511] Master 127.5.213.254:44807 requested a full tablet report, sending...
I20260618 22:39:01.898996 10098 ts_manager.cc:194] Registered new tserver with Master: f2b384901eaf487f96541c493f34ccd9 (127.5.213.193:39547)
I20260618 22:39:01.900612 10098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:52583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
I20260618 22:39:01.917878 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44807
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:41917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:02.224805 10292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:02.225059 10292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:02.225157 10292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:02.232095 10292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:02.232201 10292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:02.232321 10292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:39:02.241717 10292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:41917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:02.242944 10292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:02.244199 10292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:02.255900 10297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:02.255892 10298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:02.256006 10300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:02.256785 10292 server_base.cc:1034] running on GCE node
I20260618 22:39:02.257302 10292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:02.258154 10292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:02.259338 10292 hybrid_clock.cc:648] HybridClock initialized: now 1781822342259283 us; error 52 us; skew 500 ppm
I20260618 22:39:02.259615 10292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:02 dist-test-slave-lnj3 krb5kdc[10054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822342, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:02.262315 10292 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:39:02.263378 10292 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:02.263470 10292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:02.263749 10292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:02.266346 10292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "2b34d17c37044fb39002f9d579f805b1"
format_stamp: "Formatted at 2026-06-18 22:39:02 on dist-test-slave-lnj3"
server_key: "f9e9cfe5f08e2e3759dfec0eaff790ca"
server_key_iv: "d872127a02edfc8453bc0be3464556bb"
server_key_version: "encryptionkey@0"
I20260618 22:39:02.267114 10292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "2b34d17c37044fb39002f9d579f805b1"
format_stamp: "Formatted at 2026-06-18 22:39:02 on dist-test-slave-lnj3"
server_key: "f9e9cfe5f08e2e3759dfec0eaff790ca"
server_key_iv: "d872127a02edfc8453bc0be3464556bb"
server_key_version: "encryptionkey@0"
I20260618 22:39:02.271559 10292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260618 22:39:02.274547 10306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:02.275722 10292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:39:02.275863 10292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "2b34d17c37044fb39002f9d579f805b1"
format_stamp: "Formatted at 2026-06-18 22:39:02 on dist-test-slave-lnj3"
server_key: "f9e9cfe5f08e2e3759dfec0eaff790ca"
server_key_iv: "d872127a02edfc8453bc0be3464556bb"
server_key_version: "encryptionkey@0"
I20260618 22:39:02.276036 10292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:02.311968 10292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:02.316474 10292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:02.316735 10292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:02.317756 10292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:02.319305 10292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:02.319409 10292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:02.319510 10292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:02.319581 10292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:02.348934 10292 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:38649
I20260618 22:39:02.348984 10418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:38649 every 8 connection(s)
I20260618 22:39:02.349713 10292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:39:02.354435 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 10292
I20260618 22:39:02.354598 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:39:02.355029 5975 external_mini_cluster.cc:1428] Setting key d3c3e5cfdaa4041d73f5c62485ddbae0
Jun 18 22:39:02 dist-test-slave-lnj3 krb5kdc[10054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822342, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:39:02.369277 10288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:02.351937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:56329 (local address 127.5.213.254:44807)
0618 22:39:02.352101 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:02.352106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:02.353216 (+ 1110us) server_negotiation.cc:407] Connection header received
0618 22:39:02.354996 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:02.355002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:02.355087 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:02.355175 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:02.356866 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.357641 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.359827 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.360232 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.360320 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:02.364143 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:02.364148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:02.364151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:02.364179 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:02.366439 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:02.367221 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:02.367225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:02.367228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:02.367339 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:02.367746 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:02.367749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:02.367751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:02.367971 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:02.368067 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:02.368874 (+ 807us) server_negotiation.cc:299] Negotiation successful
0618 22:39:02.369019 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 22:39:02.370723 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:02.352377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:44807 (local address 127.5.213.194:56329)
0618 22:39:02.353053 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:02.353094 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:02.354758 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:02.355455 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:02.355466 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:02.356091 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:02.356709 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:02.356728 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.357788 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.357793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:02.359698 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:02.359711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.360426 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.360433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:02.360682 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:02.361336 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:02.361362 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:02.363944 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:02.366551 (+ 2607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:02.366558 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:02.366574 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:02.367126 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:02.367427 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:02.367432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:02.367435 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:02.367660 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:02.368083 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:02.368091 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:02.368498 (+ 407us) client_negotiation.cc:769] Sending connection context
0618 22:39:02.368799 (+ 301us) client_negotiation.cc:240] Negotiation successful
0618 22:39:02.369307 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":278,"threads_started":1}
I20260618 22:39:02.372010 10419 heartbeater.cc:347] Connected to a master server at 127.5.213.254:44807
I20260618 22:39:02.372288 10419 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.213.195@KRBTEST.COM; defaulting to no policy
I20260618 22:39:02.372895 10419 heartbeater.cc:511] Master 127.5.213.254:44807 requested a full tablet report, sending...
I20260618 22:39:02.374114 10098 ts_manager.cc:194] Registered new tserver with Master: 2b34d17c37044fb39002f9d579f805b1 (127.5.213.194:38649)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.195@KRBTEST.COM" created.
I20260618 22:39:02.374992 10098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:56329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
I20260618 22:39:02.394451 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44807
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:41917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:02.693795 10424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:02.694042 10424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:02.694186 10424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:02.701396 10424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:02.701493 10424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:02.701642 10424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:39:02.710891 10424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:41917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:02.712085 10424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:02.713349 10424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:02.725044 10432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:02.724987 10429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:02.725039 10430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:02.725826 10424 server_base.cc:1034] running on GCE node
I20260618 22:39:02.726320 10424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:02.727149 10424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:02.728353 10424 hybrid_clock.cc:648] HybridClock initialized: now 1781822342728309 us; error 31 us; skew 500 ppm
I20260618 22:39:02.728647 10424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:02 dist-test-slave-lnj3 krb5kdc[10054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822342, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:02.735044 10424 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:39:02.736040 10424 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:02.736132 10424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:02.736423 10424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:02.739130 10424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "ef0b19f6b5f34e6a9138a8b5832dee98"
format_stamp: "Formatted at 2026-06-18 22:39:02 on dist-test-slave-lnj3"
server_key: "500d429e4ee9a8f28c5980d90b35b92a"
server_key_iv: "483f9793838a25b9633c7869cf336543"
server_key_version: "encryptionkey@0"
I20260618 22:39:02.739866 10424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "ef0b19f6b5f34e6a9138a8b5832dee98"
format_stamp: "Formatted at 2026-06-18 22:39:02 on dist-test-slave-lnj3"
server_key: "500d429e4ee9a8f28c5980d90b35b92a"
server_key_iv: "483f9793838a25b9633c7869cf336543"
server_key_version: "encryptionkey@0"
I20260618 22:39:02.744344 10424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260618 22:39:02.747462 10438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:02.748593 10424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260618 22:39:02.748740 10424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "ef0b19f6b5f34e6a9138a8b5832dee98"
format_stamp: "Formatted at 2026-06-18 22:39:02 on dist-test-slave-lnj3"
server_key: "500d429e4ee9a8f28c5980d90b35b92a"
server_key_iv: "483f9793838a25b9633c7869cf336543"
server_key_version: "encryptionkey@0"
I20260618 22:39:02.748919 10424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:02.760610 10424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:02.764389 10424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:02.764614 10424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:02.765432 10424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:02.766594 10424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:02.766714 10424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:02.766793 10424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:02.766845 10424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:02.794903 10424 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:41715
I20260618 22:39:02.794958 10550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:41715 every 8 connection(s)
I20260618 22:39:02.795755 10424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:39:02.799520 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 10424
I20260618 22:39:02.799716 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:39:02.800226 5975 external_mini_cluster.cc:1428] Setting key 7a2768b464c382d8a673aaf3211f9300
Jun 18 22:39:02 dist-test-slave-lnj3 krb5kdc[10054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822342, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:39:02.814276 10288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:02.798020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:49719 (local address 127.5.213.254:44807)
0618 22:39:02.798224 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:02.798230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:02.799533 (+ 1303us) server_negotiation.cc:407] Connection header received
0618 22:39:02.801930 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:02.801935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:02.802006 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:02.802079 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:02.803725 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.804434 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.805868 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.806279 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.806365 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:02.809334 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:02.809338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:02.809342 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:02.809369 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:02.811675 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:02.812242 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:02.812246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:02.812248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:02.812346 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:02.812692 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:02.812697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:02.812699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:02.812960 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:02.813073 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:02.813797 (+ 724us) server_negotiation.cc:299] Negotiation successful
0618 22:39:02.813994 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260618 22:39:02.815405 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:02.798550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:44807 (local address 127.5.213.195:49719)
0618 22:39:02.799348 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:02.799409 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:02.801644 (+ 2235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:02.802309 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:02.802317 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:02.802918 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:02.803556 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:02.803574 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.804553 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.804557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:02.805756 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:02.805769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.806433 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.806437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:02.806610 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:02.807222 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:02.807242 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:02.809168 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:02.811776 (+ 2608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:02.811782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:02.811793 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:02.812156 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:02.812414 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:02.812417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:02.812419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:02.812570 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:02.813058 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:02.813063 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:02.813450 (+ 387us) client_negotiation.cc:769] Sending connection context
0618 22:39:02.813702 (+ 252us) client_negotiation.cc:240] Negotiation successful
0618 22:39:02.814145 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":268,"threads_started":1}
I20260618 22:39:02.816598 10551 heartbeater.cc:347] Connected to a master server at 127.5.213.254:44807
I20260618 22:39:02.816852 10551 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:02.817435 10551 heartbeater.cc:511] Master 127.5.213.254:44807 requested a full tablet report, sending...
I20260618 22:39:02.818740 10098 ts_manager.cc:194] Registered new tserver with Master: ef0b19f6b5f34e6a9138a8b5832dee98 (127.5.213.195:41715)
I20260618 22:39:02.819509 10098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:49719
I20260618 22:39:02.827004 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:39:02.841737 10288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:02.831149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.5.213.254:44807)
0618 22:39:02.831337 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:02.831344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:02.831667 (+ 323us) server_negotiation.cc:407] Connection header received
0618 22:39:02.831797 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:02.831801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:02.831854 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:02.831920 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:02.833163 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.833849 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.835101 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.835503 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.835596 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:02.837509 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:02.837514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:02.837517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:02.837543 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:02.839846 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:02.840310 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:02.840314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:02.840316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:02.840417 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:02.840676 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:02.840679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:02.840681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:02.840940 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:02.841097 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:02.841312 (+ 215us) server_negotiation.cc:299] Negotiation successful
0618 22:39:02.841445 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:39:02.845142 10098 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:39:02.851562 10098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:39:02.895957 10567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:02.862311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:38649 (local address 127.0.0.1:40878)
0618 22:39:02.887512 (+ 25201us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:02.887529 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:02.887642 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:02.888661 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:02.888669 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:02.888700 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:02.889112 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:02.889124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.890988 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.890993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:02.893383 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:02.893394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.895022 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.895031 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:02.895245 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:02.895297 (+ 52us) client_negotiation.cc:769] Sending connection context
0618 22:39:02.895433 (+ 136us) client_negotiation.cc:240] Negotiation successful
0618 22:39:02.895617 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25088,"thread_start_us":27329,"threads_started":1}
I20260618 22:39:02.895968 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:02.862562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40878 (local address 127.5.213.194:38649)
0618 22:39:02.887086 (+ 24524us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:02.887097 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:02.887754 (+ 657us) server_negotiation.cc:407] Connection header received
0618 22:39:02.887873 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:02.887882 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:02.888171 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:02.888394 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:02.889321 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.890837 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.893521 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.894795 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.894916 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:02.895124 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:02.895508 (+ 384us) server_negotiation.cc:299] Negotiation successful
0618 22:39:02.895681 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24369,"thread_start_us":24261,"threads_started":1}
I20260618 22:39:02.896937 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:02.862136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:39547 (local address 127.0.0.1:50658)
0618 22:39:02.882505 (+ 20369us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:02.882538 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:02.882726 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:02.890784 (+ 8058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:02.890788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:02.890834 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:02.891397 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:02.891407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.892634 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.892638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:02.894572 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:02.894585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.896333 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.896339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:02.896490 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:02.896513 (+ 23us) client_negotiation.cc:769] Sending connection context
0618 22:39:02.896563 (+ 50us) client_negotiation.cc:240] Negotiation successful
0618 22:39:02.896646 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20139,"mutex_wait_us":44,"spinlock_wait_cycles":1408,"thread_start_us":20001,"threads_started":1}
I20260618 22:39:02.898012 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:02.864085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.5.213.195:41715)
0618 22:39:02.889309 (+ 25224us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:02.889319 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:02.890169 (+ 850us) server_negotiation.cc:407] Connection header received
0618 22:39:02.890252 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:02.890257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:02.890436 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:02.890595 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:02.891372 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.892987 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.895318 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.896952 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.897065 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:02.897273 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:02.897671 (+ 398us) server_negotiation.cc:299] Negotiation successful
0618 22:39:02.897789 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25022,"thread_start_us":25237,"threads_started":1}
I20260618 22:39:02.898039 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:02.863785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:41715 (local address 127.0.0.1:55990)
0618 22:39:02.889917 (+ 26132us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:02.889938 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:02.890065 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:02.890606 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:02.890610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:02.890657 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:02.891134 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:02.891147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.893122 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.893127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:02.895163 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:02.895177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.897313 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.897320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:02.897475 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:02.897506 (+ 31us) client_negotiation.cc:769] Sending connection context
0618 22:39:02.897586 (+ 80us) client_negotiation.cc:240] Negotiation successful
0618 22:39:02.897701 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25994,"mutex_wait_us":4,"thread_start_us":23491,"threads_started":1}
I20260618 22:39:02.898940 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:02.862386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50658 (local address 127.5.213.193:39547)
0618 22:39:02.890219 (+ 27833us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:02.890227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:02.890256 (+ 29us) server_negotiation.cc:407] Connection header received
0618 22:39:02.890341 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:02.890348 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:02.890554 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:02.890934 (+ 380us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:02.891554 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.892506 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.894790 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.896099 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.898355 (+ 2256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:02.898508 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:02.898556 (+ 48us) server_negotiation.cc:299] Negotiation successful
0618 22:39:02.898695 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27673,"thread_start_us":27555,"threads_started":1}
I20260618 22:39:02.900209 10354 tablet_service.cc:1467] Processing CreateTablet for tablet e23795895bc54f85a139367d38da5866 (DEFAULT_TABLE table=test-table [id=ca63a50f1dcd4c98adef1155a3d0939b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:02.901503 10354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e23795895bc54f85a139367d38da5866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:02.901382 10486 tablet_service.cc:1467] Processing CreateTablet for tablet e23795895bc54f85a139367d38da5866 (DEFAULT_TABLE table=test-table [id=ca63a50f1dcd4c98adef1155a3d0939b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:02.902122 10221 tablet_service.cc:1467] Processing CreateTablet for tablet e23795895bc54f85a139367d38da5866 (DEFAULT_TABLE table=test-table [id=ca63a50f1dcd4c98adef1155a3d0939b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:02.902678 10486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e23795895bc54f85a139367d38da5866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:02.903410 10221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e23795895bc54f85a139367d38da5866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:02.903843 10286 heartbeater.cc:503] Master 127.5.213.254:44807 was elected leader, sending a full tablet report...
I20260618 22:39:02.912179 10569 tablet_bootstrap.cc:492] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1: Bootstrap starting.
I20260618 22:39:02.915822 10569 tablet_bootstrap.cc:654] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:02.916024 10570 tablet_bootstrap.cc:492] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98: Bootstrap starting.
I20260618 22:39:02.917140 10569 log.cc:824] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:02.917428 10571 tablet_bootstrap.cc:492] T e23795895bc54f85a139367d38da5866 P f2b384901eaf487f96541c493f34ccd9: Bootstrap starting.
I20260618 22:39:02.919394 10570 tablet_bootstrap.cc:654] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:02.920544 10571 tablet_bootstrap.cc:654] T e23795895bc54f85a139367d38da5866 P f2b384901eaf487f96541c493f34ccd9: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:02.920694 10570 log.cc:824] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:02.920816 10569 tablet_bootstrap.cc:492] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1: No bootstrap required, opened a new log
I20260618 22:39:02.921097 10569 ts_tablet_manager.cc:1397] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20260618 22:39:02.921623 10571 log.cc:824] T e23795895bc54f85a139367d38da5866 P f2b384901eaf487f96541c493f34ccd9: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:02.923522 10570 tablet_bootstrap.cc:492] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98: No bootstrap required, opened a new log
I20260618 22:39:02.923758 10570 ts_tablet_manager.cc:1397] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260618 22:39:02.929982 10571 tablet_bootstrap.cc:492] T e23795895bc54f85a139367d38da5866 P f2b384901eaf487f96541c493f34ccd9: No bootstrap required, opened a new log
I20260618 22:39:02.930208 10571 ts_tablet_manager.cc:1397] T e23795895bc54f85a139367d38da5866 P f2b384901eaf487f96541c493f34ccd9: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20260618 22:39:02.930904 10569 raft_consensus.cc:348] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b384901eaf487f96541c493f34ccd9" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39547 } } peers { permanent_uuid: "2b34d17c37044fb39002f9d579f805b1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38649 } } peers { permanent_uuid: "ef0b19f6b5f34e6a9138a8b5832dee98" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 41715 } }
I20260618 22:39:02.931331 10569 raft_consensus.cc:374] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:02.931428 10569 raft_consensus.cc:729] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b34d17c37044fb39002f9d579f805b1, State: Initialized, Role: FOLLOWER
I20260618 22:39:02.932000 10569 consensus_queue.cc:260] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b384901eaf487f96541c493f34ccd9" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39547 } } peers { permanent_uuid: "2b34d17c37044fb39002f9d579f805b1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38649 } } peers { permanent_uuid: "ef0b19f6b5f34e6a9138a8b5832dee98" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 41715 } }
I20260618 22:39:02.933195 10419 heartbeater.cc:503] Master 127.5.213.254:44807 was elected leader, sending a full tablet report...
I20260618 22:39:02.933681 10569 ts_tablet_manager.cc:1428] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260618 22:39:02.934062 10570 raft_consensus.cc:348] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b384901eaf487f96541c493f34ccd9" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39547 } } peers { permanent_uuid: "2b34d17c37044fb39002f9d579f805b1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38649 } } peers { permanent_uuid: "ef0b19f6b5f34e6a9138a8b5832dee98" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 41715 } }
I20260618 22:39:02.934369 10570 raft_consensus.cc:374] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:02.934455 10570 raft_consensus.cc:729] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0b19f6b5f34e6a9138a8b5832dee98, State: Initialized, Role: FOLLOWER
I20260618 22:39:02.936851 10570 consensus_queue.cc:260] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b384901eaf487f96541c493f34ccd9" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39547 } } peers { permanent_uuid: "2b34d17c37044fb39002f9d579f805b1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38649 } } peers { permanent_uuid: "ef0b19f6b5f34e6a9138a8b5832dee98" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 41715 } }
I20260618 22:39:02.938184 10577 raft_consensus.cc:482] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:39:02.938439 10577 raft_consensus.cc:504] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b384901eaf487f96541c493f34ccd9" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39547 } } peers { permanent_uuid: "2b34d17c37044fb39002f9d579f805b1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38649 } } peers { permanent_uuid: "ef0b19f6b5f34e6a9138a8b5832dee98" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 41715 } }
I20260618 22:39:02.939205 10570 ts_tablet_manager.cc:1428] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98: Time spent starting tablet: real 0.015s user 0.001s sys 0.012s
I20260618 22:39:02.939378 10571 raft_consensus.cc:348] T e23795895bc54f85a139367d38da5866 P f2b384901eaf487f96541c493f34ccd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b384901eaf487f96541c493f34ccd9" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39547 } } peers { permanent_uuid: "2b34d17c37044fb39002f9d579f805b1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38649 } } peers { permanent_uuid: "ef0b19f6b5f34e6a9138a8b5832dee98" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 41715 } }
I20260618 22:39:02.939687 10551 heartbeater.cc:503] Master 127.5.213.254:44807 was elected leader, sending a full tablet report...
I20260618 22:39:02.939839 10571 raft_consensus.cc:374] T e23795895bc54f85a139367d38da5866 P f2b384901eaf487f96541c493f34ccd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:02.939930 10571 raft_consensus.cc:729] T e23795895bc54f85a139367d38da5866 P f2b384901eaf487f96541c493f34ccd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2b384901eaf487f96541c493f34ccd9, State: Initialized, Role: FOLLOWER
I20260618 22:39:02.940371 10577 leader_election.cc:288] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2b384901eaf487f96541c493f34ccd9 (127.5.213.193:39547), 2b34d17c37044fb39002f9d579f805b1 (127.5.213.194:38649)
I20260618 22:39:02.943244 10571 consensus_queue.cc:260] T e23795895bc54f85a139367d38da5866 P f2b384901eaf487f96541c493f34ccd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b384901eaf487f96541c493f34ccd9" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39547 } } peers { permanent_uuid: "2b34d17c37044fb39002f9d579f805b1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38649 } } peers { permanent_uuid: "ef0b19f6b5f34e6a9138a8b5832dee98" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 41715 } }
I20260618 22:39:02.947695 10571 ts_tablet_manager.cc:1428] T e23795895bc54f85a139367d38da5866 P f2b384901eaf487f96541c493f34ccd9: Time spent starting tablet: real 0.017s user 0.012s sys 0.001s
I20260618 22:39:02.948185 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:02.940804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:60947 (local address 127.5.213.194:38649)
0618 22:39:02.940971 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:02.940977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:02.941428 (+ 451us) server_negotiation.cc:407] Connection header received
0618 22:39:02.941499 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:02.941504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:02.941575 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:02.941657 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:02.942097 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.943103 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.945542 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.946755 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.946904 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:02.946975 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:02.947847 (+ 872us) server_negotiation.cc:299] Negotiation successful
0618 22:39:02.947963 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:39:02.948429 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:02.940842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:38649 (local address 127.5.213.195:60947)
0618 22:39:02.941174 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:02.941194 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:02.941315 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:02.941717 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:02.941722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:02.941747 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:02.941943 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:02.941955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.943271 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.943276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:02.945348 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:02.945365 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.947068 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.947073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:02.947322 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:02.947364 (+ 42us) client_negotiation.cc:769] Sending connection context
0618 22:39:02.947455 (+ 91us) client_negotiation.cc:240] Negotiation successful
0618 22:39:02.947577 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":9}
I20260618 22:39:02.949322 10374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e23795895bc54f85a139367d38da5866" candidate_uuid: "ef0b19f6b5f34e6a9138a8b5832dee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b34d17c37044fb39002f9d579f805b1" is_pre_election: true
I20260618 22:39:02.949903 10374 raft_consensus.cc:2454] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0b19f6b5f34e6a9138a8b5832dee98 in term 0.
I20260618 22:39:02.950611 10439 leader_election.cc:302] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b34d17c37044fb39002f9d579f805b1, ef0b19f6b5f34e6a9138a8b5832dee98; no voters:
I20260618 22:39:02.951015 10577 raft_consensus.cc:2781] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:39:02.951133 10577 raft_consensus.cc:482] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:39:02.951223 10577 raft_consensus.cc:3037] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:02.952531 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:02.941123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:34401 (local address 127.5.213.193:39547)
0618 22:39:02.941277 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:02.941282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:02.946373 (+ 5091us) server_negotiation.cc:407] Connection header received
0618 22:39:02.946517 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:02.946522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:02.946589 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:02.946715 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:02.947549 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.948538 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.950690 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.951677 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.951804 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:02.951850 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:02.952236 (+ 386us) server_negotiation.cc:299] Negotiation successful
0618 22:39:02.952360 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57,"spinlock_wait_cycles":18688}
I20260618 22:39:02.952867 10578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:02.940871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:39547 (local address 127.5.213.195:34401)
0618 22:39:02.946276 (+ 5405us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:02.946290 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:02.946397 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:02.946941 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:02.946945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:02.946968 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:02.947371 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:02.947384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.948733 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.948737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:02.950554 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:02.950568 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.951878 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.951884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:02.952069 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:02.952102 (+ 33us) client_negotiation.cc:769] Sending connection context
0618 22:39:02.952221 (+ 119us) client_negotiation.cc:240] Negotiation successful
0618 22:39:02.952352 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5310,"mutex_wait_us":2,"thread_start_us":5203,"threads_started":1}
I20260618 22:39:02.953754 10241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e23795895bc54f85a139367d38da5866" candidate_uuid: "ef0b19f6b5f34e6a9138a8b5832dee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2b384901eaf487f96541c493f34ccd9" is_pre_election: true
I20260618 22:39:02.954039 10577 raft_consensus.cc:504] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b384901eaf487f96541c493f34ccd9" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39547 } } peers { permanent_uuid: "2b34d17c37044fb39002f9d579f805b1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38649 } } peers { permanent_uuid: "ef0b19f6b5f34e6a9138a8b5832dee98" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 41715 } }
I20260618 22:39:02.954210 10241 raft_consensus.cc:2454] T e23795895bc54f85a139367d38da5866 P f2b384901eaf487f96541c493f34ccd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0b19f6b5f34e6a9138a8b5832dee98 in term 0.
I20260618 22:39:02.954620 10577 leader_election.cc:288] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [CANDIDATE]: Term 1 election: Requested vote from peers f2b384901eaf487f96541c493f34ccd9 (127.5.213.193:39547), 2b34d17c37044fb39002f9d579f805b1 (127.5.213.194:38649)
I20260618 22:39:02.954957 10241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e23795895bc54f85a139367d38da5866" candidate_uuid: "ef0b19f6b5f34e6a9138a8b5832dee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2b384901eaf487f96541c493f34ccd9"
I20260618 22:39:02.955084 10374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e23795895bc54f85a139367d38da5866" candidate_uuid: "ef0b19f6b5f34e6a9138a8b5832dee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b34d17c37044fb39002f9d579f805b1"
I20260618 22:39:02.955135 10241 raft_consensus.cc:3037] T e23795895bc54f85a139367d38da5866 P f2b384901eaf487f96541c493f34ccd9 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:02.955272 10374 raft_consensus.cc:3037] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:02.957202 10241 raft_consensus.cc:2454] T e23795895bc54f85a139367d38da5866 P f2b384901eaf487f96541c493f34ccd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0b19f6b5f34e6a9138a8b5832dee98 in term 1.
I20260618 22:39:02.957453 10374 raft_consensus.cc:2454] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0b19f6b5f34e6a9138a8b5832dee98 in term 1.
I20260618 22:39:02.957521 10441 leader_election.cc:302] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ef0b19f6b5f34e6a9138a8b5832dee98, f2b384901eaf487f96541c493f34ccd9; no voters:
I20260618 22:39:02.957794 10577 raft_consensus.cc:2781] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:02.957934 10577 raft_consensus.cc:686] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [term 1 LEADER]: Becoming Leader. State: Replica: ef0b19f6b5f34e6a9138a8b5832dee98, State: Running, Role: LEADER
I20260618 22:39:02.958400 10577 consensus_queue.cc:237] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b384901eaf487f96541c493f34ccd9" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39547 } } peers { permanent_uuid: "2b34d17c37044fb39002f9d579f805b1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38649 } } peers { permanent_uuid: "ef0b19f6b5f34e6a9138a8b5832dee98" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 41715 } }
I20260618 22:39:02.962374 10098 catalog_manager.cc:5495] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0b19f6b5f34e6a9138a8b5832dee98 (127.5.213.195). New cstate: current_term: 1 leader_uuid: "ef0b19f6b5f34e6a9138a8b5832dee98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b384901eaf487f96541c493f34ccd9" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b34d17c37044fb39002f9d579f805b1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0b19f6b5f34e6a9138a8b5832dee98" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 41715 } health_report { overall_health: HEALTHY } } }
I20260618 22:39:02.997336 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:02.992355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.5.213.195:41715)
0618 22:39:02.992547 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:02.992552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:02.992569 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:39:02.992730 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:02.992734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:02.992815 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:02.992955 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:39:02.993323 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.994026 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.995363 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:02.995745 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:02.995851 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:02.996295 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:39:02.996794 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:39:02.996923 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:02.997016 (+ 93us) server_negotiation.cc:299] Negotiation successful
0618 22:39:02.997091 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 22:39:03.005092 10241 raft_consensus.cc:1261] T e23795895bc54f85a139367d38da5866 P f2b384901eaf487f96541c493f34ccd9 [term 1 FOLLOWER]: Refusing update from remote peer ef0b19f6b5f34e6a9138a8b5832dee98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:39:03.005092 10374 raft_consensus.cc:1261] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1 [term 1 FOLLOWER]: Refusing update from remote peer ef0b19f6b5f34e6a9138a8b5832dee98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:39:03.006054 10576 consensus_queue.cc:1035] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b34d17c37044fb39002f9d579f805b1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:03.006373 10577 consensus_queue.cc:1035] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2b384901eaf487f96541c493f34ccd9" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:03.017306 10582 mvcc.cc:205] Tried to move back new op lower bound from 7298344316938997760 to 7298344316761260032. Current Snapshot: MvccSnapshot[applied={T|T < 7298344316938997760}]
I20260618 22:39:03.029937 10098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:39:03.030220 10098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:39:03.033596 10098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68c949bfd1f64fbc91162efe84c12b7d: Sending DeleteTablet for 3 replicas of tablet e23795895bc54f85a139367d38da5866
I20260618 22:39:03.034708 10354 tablet_service.cc:1514] Processing DeleteTablet for tablet e23795895bc54f85a139367d38da5866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:03 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:40878
I20260618 22:39:03.034722 10221 tablet_service.cc:1514] Processing DeleteTablet for tablet e23795895bc54f85a139367d38da5866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:03 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:50658
I20260618 22:39:03.035571 10486 tablet_service.cc:1514] Processing DeleteTablet for tablet e23795895bc54f85a139367d38da5866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:03 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:55990
I20260618 22:39:03.036345 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 10159
Jun 18 22:39:03 dist-test-slave-lnj3 krb5kdc[10054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:39:03.060492 10594 tablet_replica.cc:330] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98: stopping tablet replica
I20260618 22:39:03.060968 10594 raft_consensus.cc:2229] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:39:03.061653 10594 raft_consensus.cc:2258] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:03.063485 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:03.037583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50670 (local address 127.5.213.193:39547)
0618 22:39:03.037774 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:03.037781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:03.043131 (+ 5350us) server_negotiation.cc:407] Connection header received
0618 22:39:03.043324 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:03.043330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:03.043424 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:03.043531 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:03.045274 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:03.046558 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:03.048517 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:03.049242 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:03.049337 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:03.055368 (+ 6031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:03.055388 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:03.055412 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:03.055458 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:03.059575 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:03.060438 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:03.060449 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:03.060453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:03.060756 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:03.061320 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:03.061326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:03.061330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:03.061930 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:03.062238 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:03.062834 (+ 596us) server_negotiation.cc:299] Negotiation successful
0618 22:39:03.063071 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:39:03.063784 10594 ts_tablet_manager.cc:1905] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:03.065297 10281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:03.068804 10592 tablet_replica.cc:330] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1: stopping tablet replica
I20260618 22:39:03.069209 10592 raft_consensus.cc:2229] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:39:03.069697 10592 raft_consensus.cc:2258] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:03.071457 10592 ts_tablet_manager.cc:1905] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:03.076455 10594 ts_tablet_manager.cc:1918] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:39:03.076623 10594 log.cc:1196] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/e23795895bc54f85a139367d38da5866
I20260618 22:39:03.077135 10594 ts_tablet_manager.cc:1939] T e23795895bc54f85a139367d38da5866 P ef0b19f6b5f34e6a9138a8b5832dee98: Deleting consensus metadata
I20260618 22:39:03.078092 10592 ts_tablet_manager.cc:1918] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:39:03.078227 10592 log.cc:1196] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/e23795895bc54f85a139367d38da5866
I20260618 22:39:03.078706 10592 ts_tablet_manager.cc:1939] T e23795895bc54f85a139367d38da5866 P 2b34d17c37044fb39002f9d579f805b1: Deleting consensus metadata
I20260618 22:39:03.078893 10085 catalog_manager.cc:4841] TS ef0b19f6b5f34e6a9138a8b5832dee98 (127.5.213.195:41715): tablet e23795895bc54f85a139367d38da5866 (table test-table [id=ca63a50f1dcd4c98adef1155a3d0939b]) successfully deleted
I20260618 22:39:03.080318 10082 catalog_manager.cc:4841] TS 2b34d17c37044fb39002f9d579f805b1 (127.5.213.194:38649): tablet e23795895bc54f85a139367d38da5866 (table test-table [id=ca63a50f1dcd4c98adef1155a3d0939b]) successfully deleted
I20260618 22:39:03.155452 10593 tablet_replica.cc:330] T e23795895bc54f85a139367d38da5866 P f2b384901eaf487f96541c493f34ccd9: stopping tablet replica
I20260618 22:39:03.155722 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 10159
W20260618 22:39:03.175987 10084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.213.193:39547 (error 108)
I20260618 22:39:03.176208 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 10292
I20260618 22:39:03.178164 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:03.177726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:39547 (local address 127.0.0.1:50682)
0618 22:39:03.177955 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:03.178040 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260618 22:39:03.178378 10084 catalog_manager.cc:4584] TS f2b384901eaf487f96541c493f34ccd9 (127.5.213.193:39547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e23795895bc54f85a139367d38da5866: Network error: Client connection negotiation failed: client connection to 127.5.213.193:39547: connect: Connection refused (error 111)
Jun 18 22:39:03 dist-test-slave-lnj3 krb5kdc[10054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.194@KRBTEST.COM
I20260618 22:39:03.192468 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:03.177316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40890 (local address 127.5.213.194:38649)
0618 22:39:03.177492 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:03.177498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:03.177516 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:39:03.177576 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:03.177581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:03.177653 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:03.177758 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:03.179304 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:03.180089 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:03.181939 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:03.182337 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:03.182424 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:03.187317 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:03.187330 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:03.187346 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:03.187375 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:03.189976 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:03.190744 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:03.190751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:03.190753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:03.190865 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:03.191210 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:03.191213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:03.191215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:03.191557 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:03.191761 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:03.192033 (+ 272us) server_negotiation.cc:299] Negotiation successful
0618 22:39:03.192182 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 22:39:03.193146 10414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:03.232215 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:03.231645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:39547 (local address 127.0.0.1:50690)
0618 22:39:03.231889 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:03.232022 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260618 22:39:03.233359 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:03.232988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:39547 (local address 127.0.0.1:50702)
0618 22:39:03.233137 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:03.233229 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260618 22:39:03.278103 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 10292
I20260618 22:39:03.293776 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:03.293339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:39547 (local address 127.0.0.1:50704)
0618 22:39:03.293537 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:03.293617 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260618 22:39:03.294781 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:03.294379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:39547 (local address 127.0.0.1:50716)
0618 22:39:03.294515 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:03.294595 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260618 22:39:03.296098 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 10424
Jun 18 22:39:03 dist-test-slave-lnj3 krb5kdc[10054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.195@KRBTEST.COM
I20260618 22:39:03.311091 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:03.296833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.5.213.195:41715)
0618 22:39:03.297007 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:03.297013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:03.297030 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:39:03.297160 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:03.297165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:03.297252 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:03.297348 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:03.298527 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:03.299394 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:03.300629 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:03.301134 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:03.301236 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:03.305483 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:03.305493 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:03.305511 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:03.305554 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:03.308387 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:03.308995 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:03.309005 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:03.309007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:03.309166 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:03.309510 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:03.309513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:03.309515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:03.309861 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:03.310152 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:03.310427 (+ 275us) server_negotiation.cc:299] Negotiation successful
0618 22:39:03.310711 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260618 22:39:03.311745 10546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:03.375312 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:03.374837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:39547 (local address 127.0.0.1:50728)
0618 22:39:03.375033 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:03.375137 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260618 22:39:03.376358 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:03.375952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:39547 (local address 127.0.0.1:50740)
0618 22:39:03.376158 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:03.376241 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260618 22:39:03.433187 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 10424
I20260618 22:39:03.447479 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 10067
I20260618 22:39:03.448505 10128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:03.553278 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:03.552841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:39547 (local address 127.0.0.1:50754)
0618 22:39:03.552987 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:03.553077 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:39547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260618 22:39:03.553267 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 10067
2026-06-18T22:39:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3788 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-18T22:39:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:39:03Z Disabled control of system clock
I20260618 22:39:03.610675 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:42529
--webserver_interface=127.5.213.254
--webserver_port=0
--builtin_ntp_servers=127.5.213.212:33441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:42529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260618 22:39:03.907614 10604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:03.907878 10604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:03.907984 10604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:03.915036 10604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:39:03.915128 10604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:03.915179 10604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:03.915253 10604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:39:03.915306 10604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:39:03.924235 10604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:33441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:42529
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:42529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:03.925413 10604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:03.927170 10604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:03.937752 10609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:03.937718 10610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:03.937896 10612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:03.938373 10604 server_base.cc:1034] running on GCE node
I20260618 22:39:03.939090 10604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:03.940269 10604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:03.941464 10604 hybrid_clock.cc:648] HybridClock initialized: now 1781822343941429 us; error 26 us; skew 500 ppm
I20260618 22:39:03.941784 10604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:39:03.943759 10604 webserver.cc:460] Webserver started at http://127.5.213.254:36281/ using document root <none> and password file <none>
I20260618 22:39:03.944331 10604 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:03.944410 10604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:03.944665 10604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:03.947324 10604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "94dfaf41b0f94f9a9ef003bfdfcb0f4e"
format_stamp: "Formatted at 2026-06-18 22:39:03 on dist-test-slave-lnj3"
server_key: "4c2cc138b6d70214aadca4ff1d953b92"
server_key_iv: "66215e7f2112758e92a104d0bb8a055f"
server_key_version: "encryptionkey@0"
I20260618 22:39:03.948004 10604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "94dfaf41b0f94f9a9ef003bfdfcb0f4e"
format_stamp: "Formatted at 2026-06-18 22:39:03 on dist-test-slave-lnj3"
server_key: "4c2cc138b6d70214aadca4ff1d953b92"
server_key_iv: "66215e7f2112758e92a104d0bb8a055f"
server_key_version: "encryptionkey@0"
I20260618 22:39:03.952364 10604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260618 22:39:03.955386 10618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:03.956533 10604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:39:03.956657 10604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "94dfaf41b0f94f9a9ef003bfdfcb0f4e"
format_stamp: "Formatted at 2026-06-18 22:39:03 on dist-test-slave-lnj3"
server_key: "4c2cc138b6d70214aadca4ff1d953b92"
server_key_iv: "66215e7f2112758e92a104d0bb8a055f"
server_key_version: "encryptionkey@0"
I20260618 22:39:03.956811 10604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:03.961723 10604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:03.962330 10604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:03.962553 10604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:03.977317 10604 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:42529
I20260618 22:39:03.977345 10669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:42529 every 8 connection(s)
I20260618 22:39:03.978507 10604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:39:03.981992 10670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:03.985852 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 10604
I20260618 22:39:03.986027 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:39:03.986815 5975 external_mini_cluster.cc:1428] Setting key 6606eb129cfd283e80f68ed537bf11b8
I20260618 22:39:03.994829 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:03.988987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60496 (local address 127.5.213.254:42529)
0618 22:39:03.989967 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:03.989978 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:03.990017 (+ 39us) server_negotiation.cc:407] Connection header received
0618 22:39:03.991582 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:03.991605 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:03.992008 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:03.992334 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:03.993280 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:03.993286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:03.993298 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:03.993310 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:03.993363 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:03.993378 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:03.993546 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:03.993637 (+ 91us) server_negotiation.cc:299] Negotiation successful
0618 22:39:03.993850 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":357,"threads_started":1}
I20260618 22:39:03.996119 10670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e: Bootstrap starting.
I20260618 22:39:03.998831 10670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:03.999706 10670 log.cc:824] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:04.002142 10670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e: No bootstrap required, opened a new log
I20260618 22:39:04.008956 10670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94dfaf41b0f94f9a9ef003bfdfcb0f4e" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42529 } }
I20260618 22:39:04.009270 10670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:04.009351 10670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94dfaf41b0f94f9a9ef003bfdfcb0f4e, State: Initialized, Role: FOLLOWER
I20260618 22:39:04.009868 10670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94dfaf41b0f94f9a9ef003bfdfcb0f4e" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42529 } }
I20260618 22:39:04.010067 10670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:39:04.010182 10670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:39:04.010324 10670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:04.012374 10670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94dfaf41b0f94f9a9ef003bfdfcb0f4e" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42529 } }
I20260618 22:39:04.012795 10670 leader_election.cc:302] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94dfaf41b0f94f9a9ef003bfdfcb0f4e; no voters:
I20260618 22:39:04.013306 10670 leader_election.cc:288] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:39:04.013419 10675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:04.014061 10675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e [term 1 LEADER]: Becoming Leader. State: Replica: 94dfaf41b0f94f9a9ef003bfdfcb0f4e, State: Running, Role: LEADER
I20260618 22:39:04.014516 10675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94dfaf41b0f94f9a9ef003bfdfcb0f4e" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42529 } }
I20260618 22:39:04.015204 10670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:39:04.017359 10676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94dfaf41b0f94f9a9ef003bfdfcb0f4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94dfaf41b0f94f9a9ef003bfdfcb0f4e" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42529 } } }
I20260618 22:39:04.017731 10676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:04.017401 10677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94dfaf41b0f94f9a9ef003bfdfcb0f4e. Latest consensus state: current_term: 1 leader_uuid: "94dfaf41b0f94f9a9ef003bfdfcb0f4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94dfaf41b0f94f9a9ef003bfdfcb0f4e" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42529 } } }
I20260618 22:39:04.017872 10677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:04.019066 10683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:39:04.023111 10683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:39:04.031104 10683 catalog_manager.cc:1347] Generated new cluster ID: 1d73b540188b40258e139884be471544
I20260618 22:39:04.031246 10683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:39:04.043604 10683 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:39:04.044028 10683 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:39:04.051564 10683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e: Generated new TSK 0
I20260618 22:39:04.052469 10683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:39:04.064672 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:42529
--builtin_ntp_servers=127.5.213.212:33441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260618 22:39:04.363020 10694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:04.363273 10694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:04.363392 10694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:04.370301 10694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:04.370383 10694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:04.370517 10694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:39:04.379289 10694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:33441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:42529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:04.380473 10694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:04.382157 10694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:04.393955 10700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:04.394027 10699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:04.393999 10702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:04.394716 10694 server_base.cc:1034] running on GCE node
I20260618 22:39:04.395280 10694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:04.396123 10694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:04.397310 10694 hybrid_clock.cc:648] HybridClock initialized: now 1781822344397259 us; error 47 us; skew 500 ppm
I20260618 22:39:04.397594 10694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:39:04.399739 10694 webserver.cc:460] Webserver started at http://127.5.213.193:35781/ using document root <none> and password file <none>
I20260618 22:39:04.400362 10694 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:04.400458 10694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:04.400763 10694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:04.403473 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "15d0af60687e4495b8ea3198fef29d0b"
format_stamp: "Formatted at 2026-06-18 22:39:04 on dist-test-slave-lnj3"
server_key: "f57526417016f78e7feee993da5b9547"
server_key_iv: "e321bf2315c3dc1ffd1a154a0167db50"
server_key_version: "encryptionkey@0"
I20260618 22:39:04.404203 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "15d0af60687e4495b8ea3198fef29d0b"
format_stamp: "Formatted at 2026-06-18 22:39:04 on dist-test-slave-lnj3"
server_key: "f57526417016f78e7feee993da5b9547"
server_key_iv: "e321bf2315c3dc1ffd1a154a0167db50"
server_key_version: "encryptionkey@0"
I20260618 22:39:04.409058 10694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260618 22:39:04.412185 10708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:04.413453 10694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:39:04.413601 10694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "15d0af60687e4495b8ea3198fef29d0b"
format_stamp: "Formatted at 2026-06-18 22:39:04 on dist-test-slave-lnj3"
server_key: "f57526417016f78e7feee993da5b9547"
server_key_iv: "e321bf2315c3dc1ffd1a154a0167db50"
server_key_version: "encryptionkey@0"
I20260618 22:39:04.413807 10694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:04.419333 10694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:04.420095 10694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:04.420323 10694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:04.421151 10694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:04.422389 10694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:04.422479 10694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:04.422575 10694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:04.422657 10694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:04.449410 10694 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:45645
I20260618 22:39:04.449476 10820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:45645 every 8 connection(s)
I20260618 22:39:04.450654 10694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:39:04.458072 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:04.452988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:35027 (local address 127.5.213.254:42529)
0618 22:39:04.453156 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:04.453160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:04.454491 (+ 1331us) server_negotiation.cc:407] Connection header received
0618 22:39:04.456082 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:04.456087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:04.456148 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:04.456230 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:04.457425 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:04.457432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:04.457436 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:04.457452 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:04.457480 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:04.457491 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:04.457597 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:04.457768 (+ 171us) server_negotiation.cc:299] Negotiation successful
0618 22:39:04.457828 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260618 22:39:04.458961 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:04.453491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:42529 (local address 127.5.213.193:35027)
0618 22:39:04.454341 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:04.454385 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:04.455840 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:04.456464 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:04.456471 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:04.456561 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:04.457152 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:39:04.457154 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:04.457190 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:39:04.457271 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:04.457571 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:04.457581 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:04.457588 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 22:39:04.457829 (+ 241us) client_negotiation.cc:240] Negotiation successful
0618 22:39:04.458075 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":249,"threads_started":1}
I20260618 22:39:04.459973 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 10694
I20260618 22:39:04.460130 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:39:04.460366 10821 heartbeater.cc:347] Connected to a master server at 127.5.213.254:42529
I20260618 22:39:04.460646 10821 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:04.460791 5975 external_mini_cluster.cc:1428] Setting key df5f0c6b5a3cdda455c4c3b9f071bf6d
I20260618 22:39:04.461225 10821 heartbeater.cc:511] Master 127.5.213.254:42529 requested a full tablet report, sending...
I20260618 22:39:04.463095 10635 ts_manager.cc:194] Registered new tserver with Master: 15d0af60687e4495b8ea3198fef29d0b (127.5.213.193:45645)
I20260618 22:39:04.463208 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:42529
--builtin_ntp_servers=127.5.213.212:33441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260618 22:39:04.762598 10824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:04.762889 10824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:04.763007 10824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:04.769929 10824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:04.770021 10824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:04.770154 10824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:39:04.778821 10824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:33441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:42529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:04.779963 10824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:04.781605 10824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:04.793159 10829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:04.793200 10832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:04.793220 10830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:04.794064 10824 server_base.cc:1034] running on GCE node
I20260618 22:39:04.794579 10824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:04.795411 10824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:04.796597 10824 hybrid_clock.cc:648] HybridClock initialized: now 1781822344796523 us; error 67 us; skew 500 ppm
I20260618 22:39:04.796855 10824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:39:04.798794 10824 webserver.cc:460] Webserver started at http://127.5.213.194:39453/ using document root <none> and password file <none>
I20260618 22:39:04.799371 10824 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:04.799458 10824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:04.799736 10824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:04.802273 10824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "49ee1e6b541b48a1a0a5346256a3169b"
format_stamp: "Formatted at 2026-06-18 22:39:04 on dist-test-slave-lnj3"
server_key: "9632524a934a57124a5fac46616a3eae"
server_key_iv: "32b9ab9962e5f888f121a14ad03179f9"
server_key_version: "encryptionkey@0"
I20260618 22:39:04.802997 10824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "49ee1e6b541b48a1a0a5346256a3169b"
format_stamp: "Formatted at 2026-06-18 22:39:04 on dist-test-slave-lnj3"
server_key: "9632524a934a57124a5fac46616a3eae"
server_key_iv: "32b9ab9962e5f888f121a14ad03179f9"
server_key_version: "encryptionkey@0"
I20260618 22:39:04.807512 10824 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260618 22:39:04.810562 10838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:04.811771 10824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:39:04.811897 10824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "49ee1e6b541b48a1a0a5346256a3169b"
format_stamp: "Formatted at 2026-06-18 22:39:04 on dist-test-slave-lnj3"
server_key: "9632524a934a57124a5fac46616a3eae"
server_key_iv: "32b9ab9962e5f888f121a14ad03179f9"
server_key_version: "encryptionkey@0"
I20260618 22:39:04.812083 10824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:04.817500 10824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:04.818195 10824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:04.818421 10824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:04.819339 10824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:04.820655 10824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:04.820734 10824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:04.820832 10824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:04.820894 10824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:04.849049 10824 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:40391
I20260618 22:39:04.849103 10950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:40391 every 8 connection(s)
I20260618 22:39:04.850265 10824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:39:04.857628 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:04.852610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:37159 (local address 127.5.213.254:42529)
0618 22:39:04.852795 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:04.852799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:04.854092 (+ 1293us) server_negotiation.cc:407] Connection header received
0618 22:39:04.855722 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:04.855727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:04.855777 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:04.855870 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:04.857082 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:04.857086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:04.857089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:04.857104 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:04.857132 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:04.857144 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:04.857231 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:04.857370 (+ 139us) server_negotiation.cc:299] Negotiation successful
0618 22:39:04.857415 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260618 22:39:04.858180 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 10824
I20260618 22:39:04.858317 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:39:04.858532 10953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:04.853102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:42529 (local address 127.5.213.194:37159)
0618 22:39:04.853952 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:04.853991 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:04.855454 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:04.856097 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:04.856106 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:04.856197 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:04.856810 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:39:04.856813 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:04.856849 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:39:04.856933 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:04.857215 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:04.857221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:04.857226 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:39:04.857431 (+ 205us) client_negotiation.cc:240] Negotiation successful
0618 22:39:04.857620 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":253,"threads_started":1}
I20260618 22:39:04.858793 5975 external_mini_cluster.cc:1428] Setting key bc187860b9607d386075866c4b401484
I20260618 22:39:04.859902 10951 heartbeater.cc:347] Connected to a master server at 127.5.213.254:42529
I20260618 22:39:04.860215 10951 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:04.860808 10951 heartbeater.cc:511] Master 127.5.213.254:42529 requested a full tablet report, sending...
I20260618 22:39:04.861274 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:42529
--builtin_ntp_servers=127.5.213.212:33441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260618 22:39:04.863339 10635 ts_manager.cc:194] Registered new tserver with Master: 49ee1e6b541b48a1a0a5346256a3169b (127.5.213.194:40391)
W20260618 22:39:05.163632 10954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:05.163893 10954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:05.163986 10954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:05.171041 10954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:05.171145 10954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:05.171262 10954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:39:05.180117 10954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:33441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:42529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:05.181288 10954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:05.182952 10954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:05.194655 10960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:05.194705 10959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:05.194713 10962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:05.195770 10954 server_base.cc:1034] running on GCE node
I20260618 22:39:05.196316 10954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:05.197185 10954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:05.198376 10954 hybrid_clock.cc:648] HybridClock initialized: now 1781822345198318 us; error 50 us; skew 500 ppm
I20260618 22:39:05.198700 10954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:39:05.200778 10954 webserver.cc:460] Webserver started at http://127.5.213.195:32793/ using document root <none> and password file <none>
I20260618 22:39:05.201401 10954 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:05.201495 10954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:05.201788 10954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:05.204504 10954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "39c35eafa94a4fc9acfb8e9f0ee1fa86"
format_stamp: "Formatted at 2026-06-18 22:39:05 on dist-test-slave-lnj3"
server_key: "4e0dc63cc6493d9608ecdf8b20d420b4"
server_key_iv: "fa4428b59488f183994616577ee7c005"
server_key_version: "encryptionkey@0"
I20260618 22:39:05.205303 10954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "39c35eafa94a4fc9acfb8e9f0ee1fa86"
format_stamp: "Formatted at 2026-06-18 22:39:05 on dist-test-slave-lnj3"
server_key: "4e0dc63cc6493d9608ecdf8b20d420b4"
server_key_iv: "fa4428b59488f183994616577ee7c005"
server_key_version: "encryptionkey@0"
I20260618 22:39:05.210135 10954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260618 22:39:05.213364 10968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:05.214613 10954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:39:05.214810 10954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "39c35eafa94a4fc9acfb8e9f0ee1fa86"
format_stamp: "Formatted at 2026-06-18 22:39:05 on dist-test-slave-lnj3"
server_key: "4e0dc63cc6493d9608ecdf8b20d420b4"
server_key_iv: "fa4428b59488f183994616577ee7c005"
server_key_version: "encryptionkey@0"
I20260618 22:39:05.215067 10954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:05.220647 10954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:05.221385 10954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:05.221611 10954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:05.222455 10954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:05.223874 10954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:05.223956 10954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:05.224056 10954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:05.224123 10954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:05.251689 10954 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:40117
I20260618 22:39:05.251742 11080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:40117 every 8 connection(s)
I20260618 22:39:05.252913 10954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:39:05.256320 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 10954
I20260618 22:39:05.256481 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:39:05.257005 5975 external_mini_cluster.cc:1428] Setting key 6427ec16ec6317bc22c6f5a10afe0a9e
I20260618 22:39:05.260984 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.255289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:52669 (local address 127.5.213.254:42529)
0618 22:39:05.255466 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:05.255470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:05.257066 (+ 1596us) server_negotiation.cc:407] Connection header received
0618 22:39:05.258923 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:05.258945 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:05.259010 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:05.259103 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:05.260335 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:05.260341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:05.260346 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:05.260364 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:05.260395 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:05.260406 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:05.260507 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:05.260682 (+ 175us) server_negotiation.cc:299] Negotiation successful
0618 22:39:05.260742 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:39:05.261873 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.255790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:42529 (local address 127.5.213.195:52669)
0618 22:39:05.256898 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:05.256942 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:05.258669 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:05.259339 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:05.259347 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:05.259436 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:05.260063 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:39:05.260065 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:05.260107 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:39:05.260189 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:05.260484 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:05.260494 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:05.260502 (+ 8us) client_negotiation.cc:769] Sending connection context
0618 22:39:05.260749 (+ 247us) client_negotiation.cc:240] Negotiation successful
0618 22:39:05.260992 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":289,"threads_started":1}
I20260618 22:39:05.263041 11081 heartbeater.cc:347] Connected to a master server at 127.5.213.254:42529
I20260618 22:39:05.263299 11081 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:05.263814 11081 heartbeater.cc:511] Master 127.5.213.254:42529 requested a full tablet report, sending...
I20260618 22:39:05.265029 10635 ts_manager.cc:194] Registered new tserver with Master: 39c35eafa94a4fc9acfb8e9f0ee1fa86 (127.5.213.195:40117)
I20260618 22:39:05.271682 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:39:05.278110 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.275917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.5.213.254:42529)
0618 22:39:05.276087 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:05.276094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:05.276545 (+ 451us) server_negotiation.cc:407] Connection header received
0618 22:39:05.276661 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:05.276665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:05.276714 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:05.276792 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:05.277574 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:05.277578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:05.277582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:05.277596 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:05.277627 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:05.277639 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:05.277738 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:05.277784 (+ 46us) server_negotiation.cc:299] Negotiation successful
0618 22:39:05.277826 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:39:05.281649 10635 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:39:05.287935 10635 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:39:05.305197 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.299059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41360 (local address 127.5.213.193:45645)
0618 22:39:05.300360 (+ 1301us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:05.300369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:05.301098 (+ 729us) server_negotiation.cc:407] Connection header received
0618 22:39:05.301173 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:05.301181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:05.303002 (+ 1821us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:05.303143 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:05.304395 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:05.304404 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:05.304418 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:05.304443 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:05.304497 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:05.304514 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:05.304632 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:05.304796 (+ 164us) server_negotiation.cc:299] Negotiation successful
0618 22:39:05.304888 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1149,"thread_start_us":1049,"threads_started":1}
I20260618 22:39:05.305246 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.298872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:45645 (local address 127.0.0.1:41360)
0618 22:39:05.300801 (+ 1929us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:05.300837 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:05.300967 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:05.303229 (+ 2262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:05.303236 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:05.303330 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:05.304081 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:39:05.304084 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:05.304125 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:39:05.304235 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:05.304607 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:05.304618 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:05.304625 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 22:39:05.304798 (+ 173us) client_negotiation.cc:240] Negotiation successful
0618 22:39:05.304882 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1677,"thread_start_us":1535,"threads_started":1}
I20260618 22:39:05.306007 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.301480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33990 (local address 127.5.213.195:40117)
0618 22:39:05.302914 (+ 1434us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:05.302928 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:05.302957 (+ 29us) server_negotiation.cc:407] Connection header received
0618 22:39:05.303018 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:05.303025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:05.303201 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:05.303338 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:05.305114 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:05.305123 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:05.305137 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:05.305157 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:05.305210 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:05.305229 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:05.305379 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:05.305574 (+ 195us) server_negotiation.cc:299] Negotiation successful
0618 22:39:05.305669 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1282,"thread_start_us":1122,"threads_started":1}
I20260618 22:39:05.306164 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.299004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:40391 (local address 127.0.0.1:56060)
0618 22:39:05.302126 (+ 3122us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:05.302150 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:05.302257 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:05.303993 (+ 1736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:05.303998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:05.304025 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:05.304895 (+ 870us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:39:05.304898 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:05.304911 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:39:05.304950 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:05.305426 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:05.305431 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:05.305437 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 22:39:05.305727 (+ 290us) client_negotiation.cc:240] Negotiation successful
0618 22:39:05.305773 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2975,"thread_start_us":2835,"threads_started":1}
I20260618 22:39:05.306661 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.301070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:40117 (local address 127.0.0.1:33990)
0618 22:39:05.302317 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:05.302332 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:05.302409 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:05.304025 (+ 1616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:05.304029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:05.304051 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:05.304728 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:39:05.304731 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:05.304746 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:39:05.304791 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:05.305427 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:05.305431 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:05.305437 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 22:39:05.306175 (+ 738us) client_negotiation.cc:240] Negotiation successful
0618 22:39:05.306278 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1146,"thread_start_us":1188,"threads_started":1}
I20260618 22:39:05.306775 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.299610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56060 (local address 127.5.213.194:40391)
0618 22:39:05.301684 (+ 2074us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:05.301695 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:05.302392 (+ 697us) server_negotiation.cc:407] Connection header received
0618 22:39:05.302460 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:05.302468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:05.302750 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:05.302899 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:05.305114 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:05.305123 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:05.305137 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:05.305161 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:05.305210 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:05.305229 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:05.305375 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:05.306361 (+ 986us) server_negotiation.cc:299] Negotiation successful
0618 22:39:05.306444 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1911,"thread_start_us":1760,"threads_started":1}
I20260618 22:39:05.309077 10756 tablet_service.cc:1467] Processing CreateTablet for tablet 99d0e9bffac245408d3ed634d8d33892 (DEFAULT_TABLE table=test-table [id=3d86e571554441bdb89b41b9da6118c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:05.309744 10886 tablet_service.cc:1467] Processing CreateTablet for tablet 99d0e9bffac245408d3ed634d8d33892 (DEFAULT_TABLE table=test-table [id=3d86e571554441bdb89b41b9da6118c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:05.310521 10756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d0e9bffac245408d3ed634d8d33892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:05.310894 10886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d0e9bffac245408d3ed634d8d33892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:05.310755 11016 tablet_service.cc:1467] Processing CreateTablet for tablet 99d0e9bffac245408d3ed634d8d33892 (DEFAULT_TABLE table=test-table [id=3d86e571554441bdb89b41b9da6118c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:05.312011 11016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d0e9bffac245408d3ed634d8d33892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:05.321848 11099 tablet_bootstrap.cc:492] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b: Bootstrap starting.
I20260618 22:39:05.324501 11100 tablet_bootstrap.cc:492] T 99d0e9bffac245408d3ed634d8d33892 P 15d0af60687e4495b8ea3198fef29d0b: Bootstrap starting.
I20260618 22:39:05.325070 11099 tablet_bootstrap.cc:654] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:05.325963 11101 tablet_bootstrap.cc:492] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86: Bootstrap starting.
I20260618 22:39:05.326855 11099 log.cc:824] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:05.327813 11100 tablet_bootstrap.cc:654] T 99d0e9bffac245408d3ed634d8d33892 P 15d0af60687e4495b8ea3198fef29d0b: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:05.328861 11100 log.cc:824] T 99d0e9bffac245408d3ed634d8d33892 P 15d0af60687e4495b8ea3198fef29d0b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:05.328958 11101 tablet_bootstrap.cc:654] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:05.329742 11099 tablet_bootstrap.cc:492] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b: No bootstrap required, opened a new log
I20260618 22:39:05.329963 11099 ts_tablet_manager.cc:1397] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260618 22:39:05.330027 11101 log.cc:824] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:05.331575 11100 tablet_bootstrap.cc:492] T 99d0e9bffac245408d3ed634d8d33892 P 15d0af60687e4495b8ea3198fef29d0b: No bootstrap required, opened a new log
I20260618 22:39:05.331815 11100 ts_tablet_manager.cc:1397] T 99d0e9bffac245408d3ed634d8d33892 P 15d0af60687e4495b8ea3198fef29d0b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260618 22:39:05.332269 11101 tablet_bootstrap.cc:492] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86: No bootstrap required, opened a new log
I20260618 22:39:05.332471 11101 ts_tablet_manager.cc:1397] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260618 22:39:05.339247 11101 raft_consensus.cc:348] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d0af60687e4495b8ea3198fef29d0b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45645 } } peers { permanent_uuid: "49ee1e6b541b48a1a0a5346256a3169b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40391 } } peers { permanent_uuid: "39c35eafa94a4fc9acfb8e9f0ee1fa86" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 40117 } }
I20260618 22:39:05.339579 11101 raft_consensus.cc:374] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:05.339704 11101 raft_consensus.cc:729] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c35eafa94a4fc9acfb8e9f0ee1fa86, State: Initialized, Role: FOLLOWER
I20260618 22:39:05.339787 11099 raft_consensus.cc:348] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d0af60687e4495b8ea3198fef29d0b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45645 } } peers { permanent_uuid: "49ee1e6b541b48a1a0a5346256a3169b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40391 } } peers { permanent_uuid: "39c35eafa94a4fc9acfb8e9f0ee1fa86" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 40117 } }
I20260618 22:39:05.340204 11099 raft_consensus.cc:374] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:05.340296 11099 raft_consensus.cc:729] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ee1e6b541b48a1a0a5346256a3169b, State: Initialized, Role: FOLLOWER
I20260618 22:39:05.340294 11101 consensus_queue.cc:260] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d0af60687e4495b8ea3198fef29d0b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45645 } } peers { permanent_uuid: "49ee1e6b541b48a1a0a5346256a3169b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40391 } } peers { permanent_uuid: "39c35eafa94a4fc9acfb8e9f0ee1fa86" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 40117 } }
I20260618 22:39:05.340924 11099 consensus_queue.cc:260] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d0af60687e4495b8ea3198fef29d0b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45645 } } peers { permanent_uuid: "49ee1e6b541b48a1a0a5346256a3169b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40391 } } peers { permanent_uuid: "39c35eafa94a4fc9acfb8e9f0ee1fa86" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 40117 } }
I20260618 22:39:05.341435 11100 raft_consensus.cc:348] T 99d0e9bffac245408d3ed634d8d33892 P 15d0af60687e4495b8ea3198fef29d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d0af60687e4495b8ea3198fef29d0b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45645 } } peers { permanent_uuid: "49ee1e6b541b48a1a0a5346256a3169b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40391 } } peers { permanent_uuid: "39c35eafa94a4fc9acfb8e9f0ee1fa86" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 40117 } }
I20260618 22:39:05.341907 11100 raft_consensus.cc:374] T 99d0e9bffac245408d3ed634d8d33892 P 15d0af60687e4495b8ea3198fef29d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:05.341943 11101 ts_tablet_manager.cc:1428] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260618 22:39:05.342139 11100 raft_consensus.cc:729] T 99d0e9bffac245408d3ed634d8d33892 P 15d0af60687e4495b8ea3198fef29d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d0af60687e4495b8ea3198fef29d0b, State: Initialized, Role: FOLLOWER
I20260618 22:39:05.342516 11081 heartbeater.cc:503] Master 127.5.213.254:42529 was elected leader, sending a full tablet report...
I20260618 22:39:05.342818 11100 consensus_queue.cc:260] T 99d0e9bffac245408d3ed634d8d33892 P 15d0af60687e4495b8ea3198fef29d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d0af60687e4495b8ea3198fef29d0b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45645 } } peers { permanent_uuid: "49ee1e6b541b48a1a0a5346256a3169b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40391 } } peers { permanent_uuid: "39c35eafa94a4fc9acfb8e9f0ee1fa86" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 40117 } }
I20260618 22:39:05.343869 10821 heartbeater.cc:503] Master 127.5.213.254:42529 was elected leader, sending a full tablet report...
I20260618 22:39:05.344053 10951 heartbeater.cc:503] Master 127.5.213.254:42529 was elected leader, sending a full tablet report...
I20260618 22:39:05.344393 11100 ts_tablet_manager.cc:1428] T 99d0e9bffac245408d3ed634d8d33892 P 15d0af60687e4495b8ea3198fef29d0b: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
I20260618 22:39:05.345561 11099 ts_tablet_manager.cc:1428] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
W20260618 22:39:05.351907 10952 tablet.cc:2369] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:39:05.441422 11106 raft_consensus.cc:482] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:39:05.441701 11106 raft_consensus.cc:504] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d0af60687e4495b8ea3198fef29d0b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45645 } } peers { permanent_uuid: "49ee1e6b541b48a1a0a5346256a3169b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40391 } } peers { permanent_uuid: "39c35eafa94a4fc9acfb8e9f0ee1fa86" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 40117 } }
I20260618 22:39:05.443272 11106 leader_election.cc:288] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15d0af60687e4495b8ea3198fef29d0b (127.5.213.193:45645), 39c35eafa94a4fc9acfb8e9f0ee1fa86 (127.5.213.195:40117)
I20260618 22:39:05.446372 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.443729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:55219 (local address 127.5.213.195:40117)
0618 22:39:05.443961 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:05.443968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:05.444571 (+ 603us) server_negotiation.cc:407] Connection header received
0618 22:39:05.444657 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:05.444664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:05.444723 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:05.444825 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:05.445757 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:05.445763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:05.445768 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:05.445788 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:05.445823 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:05.445849 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:05.445972 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:05.446035 (+ 63us) server_negotiation.cc:299] Negotiation successful
0618 22:39:05.446082 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260618 22:39:05.446559 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.443735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:41825 (local address 127.5.213.193:45645)
0618 22:39:05.443948 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:05.443955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:05.444685 (+ 730us) server_negotiation.cc:407] Connection header received
0618 22:39:05.444807 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:05.444811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:05.444869 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:05.444962 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:05.445966 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:05.445971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:05.445975 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:05.445993 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:05.446029 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:05.446045 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:05.446147 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:05.446271 (+ 124us) server_negotiation.cc:299] Negotiation successful
0618 22:39:05.446324 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260618 22:39:05.446547 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.443692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:40117 (local address 127.5.213.194:55219)
0618 22:39:05.444454 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:05.444470 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:05.444565 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:05.444873 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:05.444877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:05.444899 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:05.445579 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:39:05.445581 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:05.445594 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:39:05.445629 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:05.445939 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:05.445943 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:05.445948 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:39:05.446027 (+ 79us) client_negotiation.cc:240] Negotiation successful
0618 22:39:05.446075 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":505,"threads_started":1}
I20260618 22:39:05.447074 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.443745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:45645 (local address 127.5.213.194:41825)
0618 22:39:05.444569 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:05.444588 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:05.444701 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:05.445012 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:05.445016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:05.445046 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:05.445757 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:39:05.445760 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:05.445777 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:39:05.445820 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:05.446142 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:05.446148 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:05.446153 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:39:05.446307 (+ 154us) client_negotiation.cc:240] Negotiation successful
0618 22:39:05.446355 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":484,"threads_started":1}
I20260618 22:39:05.447649 11036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d0e9bffac245408d3ed634d8d33892" candidate_uuid: "49ee1e6b541b48a1a0a5346256a3169b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c35eafa94a4fc9acfb8e9f0ee1fa86" is_pre_election: true
I20260618 22:39:05.447912 10776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d0e9bffac245408d3ed634d8d33892" candidate_uuid: "49ee1e6b541b48a1a0a5346256a3169b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d0af60687e4495b8ea3198fef29d0b" is_pre_election: true
I20260618 22:39:05.448084 11036 raft_consensus.cc:2454] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49ee1e6b541b48a1a0a5346256a3169b in term 0.
I20260618 22:39:05.448406 10776 raft_consensus.cc:2454] T 99d0e9bffac245408d3ed634d8d33892 P 15d0af60687e4495b8ea3198fef29d0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49ee1e6b541b48a1a0a5346256a3169b in term 0.
I20260618 22:39:05.448673 10841 leader_election.cc:302] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c35eafa94a4fc9acfb8e9f0ee1fa86, 49ee1e6b541b48a1a0a5346256a3169b; no voters:
I20260618 22:39:05.449040 11106 raft_consensus.cc:2781] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:39:05.449201 11106 raft_consensus.cc:482] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:39:05.449297 11106 raft_consensus.cc:3037] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:05.451552 11106 raft_consensus.cc:504] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d0af60687e4495b8ea3198fef29d0b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45645 } } peers { permanent_uuid: "49ee1e6b541b48a1a0a5346256a3169b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40391 } } peers { permanent_uuid: "39c35eafa94a4fc9acfb8e9f0ee1fa86" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 40117 } }
I20260618 22:39:05.452145 11106 leader_election.cc:288] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [CANDIDATE]: Term 1 election: Requested vote from peers 15d0af60687e4495b8ea3198fef29d0b (127.5.213.193:45645), 39c35eafa94a4fc9acfb8e9f0ee1fa86 (127.5.213.195:40117)
I20260618 22:39:05.452590 11036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d0e9bffac245408d3ed634d8d33892" candidate_uuid: "49ee1e6b541b48a1a0a5346256a3169b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c35eafa94a4fc9acfb8e9f0ee1fa86"
W20260618 22:39:05.452627 10822 tablet.cc:2369] T 99d0e9bffac245408d3ed634d8d33892 P 15d0af60687e4495b8ea3198fef29d0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:39:05.452620 10776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d0e9bffac245408d3ed634d8d33892" candidate_uuid: "49ee1e6b541b48a1a0a5346256a3169b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d0af60687e4495b8ea3198fef29d0b"
I20260618 22:39:05.452769 11036 raft_consensus.cc:3037] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:05.452828 10776 raft_consensus.cc:3037] T 99d0e9bffac245408d3ed634d8d33892 P 15d0af60687e4495b8ea3198fef29d0b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:05.455446 10776 raft_consensus.cc:2454] T 99d0e9bffac245408d3ed634d8d33892 P 15d0af60687e4495b8ea3198fef29d0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ee1e6b541b48a1a0a5346256a3169b in term 1.
I20260618 22:39:05.455775 11036 raft_consensus.cc:2454] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ee1e6b541b48a1a0a5346256a3169b in term 1.
I20260618 22:39:05.455878 10839 leader_election.cc:302] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d0af60687e4495b8ea3198fef29d0b, 49ee1e6b541b48a1a0a5346256a3169b; no voters:
I20260618 22:39:05.456219 11106 raft_consensus.cc:2781] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:05.456727 11106 raft_consensus.cc:686] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [term 1 LEADER]: Becoming Leader. State: Replica: 49ee1e6b541b48a1a0a5346256a3169b, State: Running, Role: LEADER
I20260618 22:39:05.457194 11106 consensus_queue.cc:237] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d0af60687e4495b8ea3198fef29d0b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45645 } } peers { permanent_uuid: "49ee1e6b541b48a1a0a5346256a3169b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40391 } } peers { permanent_uuid: "39c35eafa94a4fc9acfb8e9f0ee1fa86" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 40117 } }
I20260618 22:39:05.461767 10634 catalog_manager.cc:5495] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b reported cstate change: term changed from 0 to 1, leader changed from <none> to 49ee1e6b541b48a1a0a5346256a3169b (127.5.213.194). New cstate: current_term: 1 leader_uuid: "49ee1e6b541b48a1a0a5346256a3169b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d0af60687e4495b8ea3198fef29d0b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49ee1e6b541b48a1a0a5346256a3169b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39c35eafa94a4fc9acfb8e9f0ee1fa86" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20260618 22:39:05.502141 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.500160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56072 (local address 127.5.213.194:40391)
0618 22:39:05.500350 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:05.500356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:05.500375 (+ 19us) server_negotiation.cc:407] Connection header received
0618 22:39:05.500570 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:05.500574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:05.500655 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:05.500740 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:05.501572 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:05.501577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:05.501582 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:05.501597 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:05.501628 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:05.501640 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:05.501744 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:05.501852 (+ 108us) server_negotiation.cc:299] Negotiation successful
0618 22:39:05.501908 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260618 22:39:05.504338 11082 tablet.cc:2369] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:39:05.510186 11036 raft_consensus.cc:1261] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86 [term 1 FOLLOWER]: Refusing update from remote peer 49ee1e6b541b48a1a0a5346256a3169b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:39:05.510191 10776 raft_consensus.cc:1261] T 99d0e9bffac245408d3ed634d8d33892 P 15d0af60687e4495b8ea3198fef29d0b [term 1 FOLLOWER]: Refusing update from remote peer 49ee1e6b541b48a1a0a5346256a3169b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:39:05.511133 11106 consensus_queue.cc:1035] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [LEADER]: Connected to new peer: Peer: permanent_uuid: "39c35eafa94a4fc9acfb8e9f0ee1fa86" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:05.511407 11110 consensus_queue.cc:1035] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [LEADER]: Connected to new peer: Peer: permanent_uuid: "15d0af60687e4495b8ea3198fef29d0b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:05.540836 10634 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:39:05.541105 10634 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:39:05.544679 10634 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94dfaf41b0f94f9a9ef003bfdfcb0f4e: Sending DeleteTablet for 3 replicas of tablet 99d0e9bffac245408d3ed634d8d33892
I20260618 22:39:05.545846 11016 tablet_service.cc:1514] Processing DeleteTablet for tablet 99d0e9bffac245408d3ed634d8d33892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:05 UTC) from {username='slave'} at 127.0.0.1:33990
I20260618 22:39:05.545861 10886 tablet_service.cc:1514] Processing DeleteTablet for tablet 99d0e9bffac245408d3ed634d8d33892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:05 UTC) from {username='slave'} at 127.0.0.1:56060
I20260618 22:39:05.545841 10756 tablet_service.cc:1514] Processing DeleteTablet for tablet 99d0e9bffac245408d3ed634d8d33892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:05 UTC) from {username='slave'} at 127.0.0.1:41360
I20260618 22:39:05.549585 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 10694
I20260618 22:39:05.556521 11125 tablet_replica.cc:330] T 99d0e9bffac245408d3ed634d8d33892 P 15d0af60687e4495b8ea3198fef29d0b: stopping tablet replica
I20260618 22:39:05.557085 11123 tablet_replica.cc:330] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86: stopping tablet replica
I20260618 22:39:05.557098 11125 raft_consensus.cc:2229] T 99d0e9bffac245408d3ed634d8d33892 P 15d0af60687e4495b8ea3198fef29d0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:39:05.557688 11125 raft_consensus.cc:2258] T 99d0e9bffac245408d3ed634d8d33892 P 15d0af60687e4495b8ea3198fef29d0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:05.557751 11123 raft_consensus.cc:2229] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:39:05.558485 11124 tablet_replica.cc:330] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b: stopping tablet replica
I20260618 22:39:05.558946 11124 raft_consensus.cc:2229] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:39:05.559648 11124 raft_consensus.cc:2258] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:05.559901 11125 ts_tablet_manager.cc:1905] T 99d0e9bffac245408d3ed634d8d33892 P 15d0af60687e4495b8ea3198fef29d0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:05.558182 11123 raft_consensus.cc:2258] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:05.563611 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.551218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41366 (local address 127.5.213.193:45645)
0618 22:39:05.551410 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:05.551417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:05.561288 (+ 9871us) server_negotiation.cc:407] Connection header received
0618 22:39:05.561552 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:05.561558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:05.561649 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:05.561724 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:05.562896 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:05.562902 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:05.562907 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:05.562925 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:05.562960 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:05.562974 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:05.563060 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:05.563276 (+ 216us) server_negotiation.cc:299] Negotiation successful
0618 22:39:05.563334 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260618 22:39:05.563868 11123 ts_tablet_manager.cc:1905] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:05.564745 10816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:05.565716 11124 ts_tablet_manager.cc:1905] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:05.571672 11123 ts_tablet_manager.cc:1918] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:39:05.571960 11123 log.cc:1196] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/99d0e9bffac245408d3ed634d8d33892
I20260618 22:39:05.572569 11123 ts_tablet_manager.cc:1939] T 99d0e9bffac245408d3ed634d8d33892 P 39c35eafa94a4fc9acfb8e9f0ee1fa86: Deleting consensus metadata
I20260618 22:39:05.572729 11124 ts_tablet_manager.cc:1918] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:39:05.572942 11124 log.cc:1196] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/99d0e9bffac245408d3ed634d8d33892
I20260618 22:39:05.573503 11124 ts_tablet_manager.cc:1939] T 99d0e9bffac245408d3ed634d8d33892 P 49ee1e6b541b48a1a0a5346256a3169b: Deleting consensus metadata
I20260618 22:39:05.574807 10621 catalog_manager.cc:4841] TS 39c35eafa94a4fc9acfb8e9f0ee1fa86 (127.5.213.195:40117): tablet 99d0e9bffac245408d3ed634d8d33892 (table test-table [id=3d86e571554441bdb89b41b9da6118c3]) successfully deleted
I20260618 22:39:05.575168 10620 catalog_manager.cc:4841] TS 49ee1e6b541b48a1a0a5346256a3169b (127.5.213.194:40391): tablet 99d0e9bffac245408d3ed634d8d33892 (table test-table [id=3d86e571554441bdb89b41b9da6118c3]) successfully deleted
I20260618 22:39:05.660061 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 10694
I20260618 22:39:05.681012 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 10824
W20260618 22:39:05.681071 10619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.213.193:45645 (error 108)
I20260618 22:39:05.684166 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.683669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:45645 (local address 127.0.0.1:41376)
0618 22:39:05.683861 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:05.683994 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260618 22:39:05.684345 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.682123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.5.213.194:40391)
0618 22:39:05.682327 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:05.682334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:05.682355 (+ 21us) server_negotiation.cc:407] Connection header received
0618 22:39:05.682452 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:05.682458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:05.682529 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:05.682674 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:05.683603 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:05.683610 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:05.683618 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:05.683637 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:05.683674 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:05.683689 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:05.683815 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:05.684007 (+ 192us) server_negotiation.cc:299] Negotiation successful
0618 22:39:05.684059 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260618 22:39:05.684403 10619 catalog_manager.cc:4584] TS 15d0af60687e4495b8ea3198fef29d0b (127.5.213.193:45645): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 99d0e9bffac245408d3ed634d8d33892: Network error: Client connection negotiation failed: client connection to 127.5.213.193:45645: connect: Connection refused (error 111)
I20260618 22:39:05.684984 10946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:05.737892 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.737358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:45645 (local address 127.0.0.1:41388)
0618 22:39:05.737572 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:05.737690 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260618 22:39:05.738948 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.738556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:45645 (local address 127.0.0.1:41392)
0618 22:39:05.738723 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:05.738816 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260618 22:39:05.790715 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 10824
I20260618 22:39:05.799528 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.798997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:45645 (local address 127.0.0.1:41398)
0618 22:39:05.799222 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:05.799331 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260618 22:39:05.800493 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.800127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:45645 (local address 127.0.0.1:41414)
0618 22:39:05.800280 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:05.800364 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260618 22:39:05.805085 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 10954
I20260618 22:39:05.807478 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.805707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33996 (local address 127.5.213.195:40117)
0618 22:39:05.805884 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:05.805889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:05.805910 (+ 21us) server_negotiation.cc:407] Connection header received
0618 22:39:05.806095 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:05.806099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:05.806171 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:05.806249 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:05.807040 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:05.807044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:05.807047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:05.807060 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:05.807089 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:05.807100 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:05.807195 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:05.807239 (+ 44us) server_negotiation.cc:299] Negotiation successful
0618 22:39:05.807282 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260618 22:39:05.808032 11076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:05.880923 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.880491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:45645 (local address 127.0.0.1:41426)
0618 22:39:05.880692 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:05.880790 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260618 22:39:05.881927 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:05.881561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:45645 (local address 127.0.0.1:41440)
0618 22:39:05.881742 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:05.881809 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260618 22:39:05.891964 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 10954
I20260618 22:39:05.904192 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 10604
I20260618 22:39:05.905256 10665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:06.012856 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 10604
2026-06-18T22:39:06Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2458 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260618 22:39:06.048723 5975 mini_oidc.cc:152] Starting JWKS server
I20260618 22:39:06.051373 5975 webserver.cc:460] Webserver started at https://127.0.0.1:39377/ using document root <none> and password file <none>
I20260618 22:39:06.051643 5975 mini_oidc.cc:189] Starting OIDC Discovery server
I20260618 22:39:06.052215 5975 webserver.cc:460] Webserver started at http://127.0.0.1:41579/ using document root <none> and password file <none>
2026-06-18T22:39:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:39:06Z Disabled control of system clock
I20260618 22:39:06.075254 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:33901
--webserver_interface=127.5.213.254
--webserver_port=0
--builtin_ntp_servers=127.5.213.212:37935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:33901
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39377/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781822307836422-5975-0/testchainca.cert with env {}
W20260618 22:39:06.395430 11137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:06.395702 11137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:06.395797 11137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:06.402719 11137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:39:06.402817 11137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:06.402855 11137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:06.402909 11137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:39:06.402951 11137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260618 22:39:06.403007 11137 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260618 22:39:06.403046 11137 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39377/jwks.json
I20260618 22:39:06.412542 11137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:33901
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:33901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39377/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781822307836422-5975-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:06.413743 11137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:06.414973 11137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:06.425555 11142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:06.425597 11145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:06.425627 11143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:06.426272 11137 server_base.cc:1034] running on GCE node
I20260618 22:39:06.427109 11137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:06.428433 11137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:06.429669 11137 hybrid_clock.cc:648] HybridClock initialized: now 1781822346429606 us; error 50 us; skew 500 ppm
I20260618 22:39:06.430011 11137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:39:06.432245 11137 webserver.cc:460] Webserver started at http://127.5.213.254:32937/ using document root <none> and password file <none>
I20260618 22:39:06.432893 11137 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:06.432977 11137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:06.433219 11137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:06.436085 11137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "3e5e661d76584a68ae551796c9f5f177"
format_stamp: "Formatted at 2026-06-18 22:39:06 on dist-test-slave-lnj3"
server_key: "3a93636dcca6a42600c30ce973a724e6"
server_key_iv: "a229852643d29a80a3bbe8c83d64a8b3"
server_key_version: "encryptionkey@0"
I20260618 22:39:06.436828 11137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "3e5e661d76584a68ae551796c9f5f177"
format_stamp: "Formatted at 2026-06-18 22:39:06 on dist-test-slave-lnj3"
server_key: "3a93636dcca6a42600c30ce973a724e6"
server_key_iv: "a229852643d29a80a3bbe8c83d64a8b3"
server_key_version: "encryptionkey@0"
I20260618 22:39:06.441660 11137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260618 22:39:06.444872 11151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:06.446197 11137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260618 22:39:06.446363 11137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "3e5e661d76584a68ae551796c9f5f177"
format_stamp: "Formatted at 2026-06-18 22:39:06 on dist-test-slave-lnj3"
server_key: "3a93636dcca6a42600c30ce973a724e6"
server_key_iv: "a229852643d29a80a3bbe8c83d64a8b3"
server_key_version: "encryptionkey@0"
I20260618 22:39:06.446558 11137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:06.462369 11137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:06.463204 11137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:06.463479 11137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:06.479434 11137 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:33901
I20260618 22:39:06.479452 11202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:33901 every 8 connection(s)
I20260618 22:39:06.480747 11137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:39:06.481392 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 11137
I20260618 22:39:06.481606 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:39:06.482278 5975 external_mini_cluster.cc:1428] Setting key 10b94947e68c8e0c2ae926c3598d0ecc
I20260618 22:39:06.485179 11203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:06.495398 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:06.484047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55286 (local address 127.5.213.254:33901)
0618 22:39:06.484952 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:06.484968 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:06.485007 (+ 39us) server_negotiation.cc:407] Connection header received
0618 22:39:06.486563 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:06.486585 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:06.487059 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:06.487378 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:06.488006 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:06.489390 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:06.491190 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:06.491807 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:06.491936 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:06.493136 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:06.493143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:06.493156 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:06.493704 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:06.493776 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:06.493797 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:06.493888 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:06.494120 (+ 232us) server_negotiation.cc:299] Negotiation successful
0618 22:39:06.494434 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":302,"threads_started":1}
I20260618 22:39:06.496704 11203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177: Bootstrap starting.
I20260618 22:39:06.499179 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:39:06.500203 11203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:06.501330 11203 log.cc:824] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:06.504109 11203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177: No bootstrap required, opened a new log
I20260618 22:39:06.511161 11203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5e661d76584a68ae551796c9f5f177" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33901 } }
I20260618 22:39:06.511479 11203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:06.511549 11203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e5e661d76584a68ae551796c9f5f177, State: Initialized, Role: FOLLOWER
I20260618 22:39:06.512063 11203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5e661d76584a68ae551796c9f5f177" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33901 } }
I20260618 22:39:06.512264 11203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:39:06.512356 11203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:39:06.512478 11203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:06.514660 11203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5e661d76584a68ae551796c9f5f177" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33901 } }
I20260618 22:39:06.515206 11203 leader_election.cc:302] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e5e661d76584a68ae551796c9f5f177; no voters:
I20260618 22:39:06.515784 11203 leader_election.cc:288] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:39:06.515902 11208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:06.516530 11208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177 [term 1 LEADER]: Becoming Leader. State: Replica: 3e5e661d76584a68ae551796c9f5f177, State: Running, Role: LEADER
I20260618 22:39:06.517027 11208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5e661d76584a68ae551796c9f5f177" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33901 } }
I20260618 22:39:06.517616 11203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:39:06.519598 11209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e5e661d76584a68ae551796c9f5f177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5e661d76584a68ae551796c9f5f177" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33901 } } }
I20260618 22:39:06.519994 11209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:06.520171 11210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e5e661d76584a68ae551796c9f5f177. Latest consensus state: current_term: 1 leader_uuid: "3e5e661d76584a68ae551796c9f5f177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5e661d76584a68ae551796c9f5f177" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33901 } } }
I20260618 22:39:06.520449 11210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:06.521991 11217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:39:06.525987 11217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:39:06.534063 11217 catalog_manager.cc:1347] Generated new cluster ID: 5e7abfc4ea8b444386f4bd4d987962e9
I20260618 22:39:06.534171 11217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:39:06.552443 11217 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:39:06.554242 11217 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:39:06.564116 11217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e5e661d76584a68ae551796c9f5f177: Generated new TSK 0
I20260618 22:39:06.565197 11217 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260618 22:39:06.608247 11233 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260618 22:39:06.608412 11233 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260618 22:39:06.609587 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:06.587910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55288 (local address 127.5.213.254:33901)
0618 22:39:06.588106 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:06.588113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:06.588576 (+ 463us) server_negotiation.cc:407] Connection header received
0618 22:39:06.588866 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:06.588872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:06.588973 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:06.589134 (+ 161us) server_negotiation.cc:226] Negotiated authn=JWT
0618 22:39:06.589602 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:06.590765 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:06.592184 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:06.592870 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:06.592987 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:06.593518 (+ 531us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0618 22:39:06.608875 (+ 15357us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0618 22:39:06.609277 (+ 402us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55288: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260618 22:39:06.609711 11206 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55288: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260618 22:39:06.611181 11232 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:06.587828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33901 (local address 127.0.0.1:55288)
0618 22:39:06.588482 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:06.588498 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:06.588656 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:06.589161 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:06.589165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:06.589175 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0618 22:39:06.589463 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:06.589471 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:06.590926 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:06.590930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:06.592042 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:06.592053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:06.593022 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:06.593026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:06.593164 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:06.593226 (+ 62us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0618 22:39:06.609184 (+ 15958us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0618 22:39:06.609460 (+ 276us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.213.254:33901: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":342,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.5.213.254:33901: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39377/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260618 22:39:06.612797 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 11137
I20260618 22:39:06.613993 11198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:06.704612 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 11137
2026-06-18T22:39:06Z chronyd exiting
I20260618 22:39:06.750803 5975 test_util.cc:175] -----------------------------------------------
I20260618 22:39:06.750949 5975 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781822307836422-5975-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (706 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260618 22:39:06.754468 5975 mini_oidc.cc:152] Starting JWKS server
I20260618 22:39:06.756290 5975 webserver.cc:460] Webserver started at https://127.0.0.1:37283/ using document root <none> and password file <none>
I20260618 22:39:06.756546 5975 mini_oidc.cc:189] Starting OIDC Discovery server
I20260618 22:39:06.757138 5975 webserver.cc:460] Webserver started at http://127.0.0.1:41659/ using document root <none> and password file <none>
2026-06-18T22:39:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:39:06Z Disabled control of system clock
I20260618 22:39:06.779919 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:37563
--webserver_interface=127.5.213.254
--webserver_port=0
--builtin_ntp_servers=127.5.213.212:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:37563
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37283/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781822307836422-5975-0/testchainca.cert with env {}
W20260618 22:39:07.087479 11240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:07.087771 11240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:07.087906 11240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:07.094951 11240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:39:07.095074 11240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:07.095142 11240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:07.095232 11240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:39:07.095291 11240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260618 22:39:07.095381 11240 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260618 22:39:07.095432 11240 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37283/jwks.json
I20260618 22:39:07.105113 11240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:37563
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:37563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37283/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781822307836422-5975-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:07.106308 11240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:07.107599 11240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:07.118592 11245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:07.118695 11246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:07.118655 11248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:07.119437 11240 server_base.cc:1034] running on GCE node
I20260618 22:39:07.120200 11240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:07.121559 11240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:07.122735 11240 hybrid_clock.cc:648] HybridClock initialized: now 1781822347122666 us; error 51 us; skew 500 ppm
I20260618 22:39:07.123092 11240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:39:07.125304 11240 webserver.cc:460] Webserver started at http://127.5.213.254:40139/ using document root <none> and password file <none>
I20260618 22:39:07.125967 11240 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:07.126075 11240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:07.126389 11240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:07.129221 11240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "a7e3def7226d4f868387704959c4cb7b"
format_stamp: "Formatted at 2026-06-18 22:39:07 on dist-test-slave-lnj3"
server_key: "2acbc052be5a6902b2aa25852c0ae5ea"
server_key_iv: "b6ce2eb6d7ed1deb84a8558e22551ef6"
server_key_version: "encryptionkey@0"
I20260618 22:39:07.129979 11240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "a7e3def7226d4f868387704959c4cb7b"
format_stamp: "Formatted at 2026-06-18 22:39:07 on dist-test-slave-lnj3"
server_key: "2acbc052be5a6902b2aa25852c0ae5ea"
server_key_iv: "b6ce2eb6d7ed1deb84a8558e22551ef6"
server_key_version: "encryptionkey@0"
I20260618 22:39:07.134966 11240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260618 22:39:07.138217 11254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:07.139531 11240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:39:07.139695 11240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "a7e3def7226d4f868387704959c4cb7b"
format_stamp: "Formatted at 2026-06-18 22:39:07 on dist-test-slave-lnj3"
server_key: "2acbc052be5a6902b2aa25852c0ae5ea"
server_key_iv: "b6ce2eb6d7ed1deb84a8558e22551ef6"
server_key_version: "encryptionkey@0"
I20260618 22:39:07.139882 11240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:07.166091 11240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:07.166961 11240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:07.167273 11240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:07.184734 11240 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:37563
I20260618 22:39:07.184768 11305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:37563 every 8 connection(s)
I20260618 22:39:07.186077 11240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:39:07.189569 11306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:07.193934 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 11240
I20260618 22:39:07.194126 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:39:07.194921 5975 external_mini_cluster.cc:1428] Setting key 00e1ea789470432898800faf0620cfc0
I20260618 22:39:07.200218 11306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b: Bootstrap starting.
I20260618 22:39:07.203450 11306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:07.204497 11306 log.cc:824] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:07.207363 11306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b: No bootstrap required, opened a new log
I20260618 22:39:07.207543 11309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:07.196683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.5.213.254:37563)
0618 22:39:07.197589 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:07.197603 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:07.197645 (+ 42us) server_negotiation.cc:407] Connection header received
0618 22:39:07.199293 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:07.199313 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:07.199735 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:07.200023 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:07.200564 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:07.201804 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:07.203531 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:07.204187 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:07.204313 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:07.205392 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:07.205402 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:07.205418 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:07.205994 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:07.206068 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:07.206087 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:07.206163 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:07.206389 (+ 226us) server_negotiation.cc:299] Negotiation successful
0618 22:39:07.206681 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":339,"threads_started":1}
I20260618 22:39:07.211176 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:39:07.216128 11306 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e3def7226d4f868387704959c4cb7b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 37563 } }
I20260618 22:39:07.216447 11306 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:07.216550 11306 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7e3def7226d4f868387704959c4cb7b, State: Initialized, Role: FOLLOWER
I20260618 22:39:07.217098 11306 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e3def7226d4f868387704959c4cb7b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 37563 } }
I20260618 22:39:07.217300 11306 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:39:07.217422 11306 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:39:07.217559 11306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:07.219565 11306 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e3def7226d4f868387704959c4cb7b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 37563 } }
I20260618 22:39:07.220016 11306 leader_election.cc:302] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7e3def7226d4f868387704959c4cb7b; no voters:
I20260618 22:39:07.220556 11306 leader_election.cc:288] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:39:07.220669 11311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:07.221450 11311 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b [term 1 LEADER]: Becoming Leader. State: Replica: a7e3def7226d4f868387704959c4cb7b, State: Running, Role: LEADER
I20260618 22:39:07.221931 11311 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e3def7226d4f868387704959c4cb7b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 37563 } }
I20260618 22:39:07.222610 11306 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:39:07.224576 11313 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7e3def7226d4f868387704959c4cb7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e3def7226d4f868387704959c4cb7b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 37563 } } }
I20260618 22:39:07.225055 11313 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:07.224604 11312 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7e3def7226d4f868387704959c4cb7b. Latest consensus state: current_term: 1 leader_uuid: "a7e3def7226d4f868387704959c4cb7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e3def7226d4f868387704959c4cb7b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 37563 } } }
I20260618 22:39:07.225456 11312 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:07.227613 11317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:39:07.232836 11317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:39:07.241346 11317 catalog_manager.cc:1347] Generated new cluster ID: 346c4217af0e4b73b0067a657238add8
I20260618 22:39:07.241433 11317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:39:07.261758 11317 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:39:07.263563 11317 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:39:07.278430 11317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7e3def7226d4f868387704959c4cb7b: Generated new TSK 0
I20260618 22:39:07.279510 11317 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260618 22:39:07.317865 11336 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260618 22:39:07.317970 11336 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260618 22:39:07.319257 11309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:07.299306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.5.213.254:37563)
0618 22:39:07.299487 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:07.299493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:07.299825 (+ 332us) server_negotiation.cc:407] Connection header received
0618 22:39:07.300016 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:07.300022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:07.300127 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:07.300308 (+ 181us) server_negotiation.cc:226] Negotiated authn=JWT
0618 22:39:07.300686 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:07.301871 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:07.303223 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:07.303896 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:07.304024 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:07.304444 (+ 420us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0618 22:39:07.318563 (+ 14119us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37283/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0618 22:39:07.318952 (+ 389us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32850: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37283/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":68}
W20260618 22:39:07.319276 11335 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:07.299160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:37563 (local address 127.0.0.1:32850)
0618 22:39:07.299722 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:07.299738 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:07.299861 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:07.300311 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:07.300315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:07.300323 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0618 22:39:07.300544 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:07.300551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:07.301994 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:07.301997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:07.303097 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:07.303109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:07.304050 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:07.304054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:07.304189 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:07.304219 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0618 22:39:07.318812 (+ 14593us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37283/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0618 22:39:07.319001 (+ 189us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.213.254:37563: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37283/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":296,"threads_started":1}
W20260618 22:39:07.319375 11309 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32850: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37283/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260618 22:39:07.319916 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 11240
I20260618 22:39:07.320953 11301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:07.411348 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 11240
2026-06-18T22:39:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (707 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260618 22:39:07.461961 5975 mini_oidc.cc:152] Starting JWKS server
I20260618 22:39:07.463718 5975 webserver.cc:460] Webserver started at https://127.0.0.1:35073/ using document root <none> and password file <none>
I20260618 22:39:07.463915 5975 mini_oidc.cc:189] Starting OIDC Discovery server
I20260618 22:39:07.464476 5975 webserver.cc:460] Webserver started at http://127.0.0.1:34697/ using document root <none> and password file <none>
2026-06-18T22:39:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:39:07Z Disabled control of system clock
I20260618 22:39:07.486882 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:39475
--webserver_interface=127.5.213.254
--webserver_port=0
--builtin_ntp_servers=127.5.213.212:38343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:39475
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35073/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260618 22:39:07.788213 11343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:07.788513 11343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:07.788648 11343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:07.795710 11343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:39:07.795840 11343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:07.795933 11343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:07.796012 11343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:39:07.796074 11343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260618 22:39:07.796154 11343 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260618 22:39:07.796201 11343 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35073/jwks.json
I20260618 22:39:07.805219 11343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:38343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:39475
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:39475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35073/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:07.806486 11343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:07.808231 11343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:07.819315 11351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:07.819279 11348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:07.819362 11349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:07.828305 11343 server_base.cc:1034] running on GCE node
I20260618 22:39:07.829125 11343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:07.830658 11343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:07.831880 11343 hybrid_clock.cc:648] HybridClock initialized: now 1781822347831838 us; error 35 us; skew 500 ppm
I20260618 22:39:07.832223 11343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:39:07.839584 11343 webserver.cc:460] Webserver started at http://127.5.213.254:44735/ using document root <none> and password file <none>
I20260618 22:39:07.840293 11343 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:07.840399 11343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:07.840701 11343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:07.843583 11343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "820b4c77e26a4d16a9991df5a8432bdf"
format_stamp: "Formatted at 2026-06-18 22:39:07 on dist-test-slave-lnj3"
server_key: "4948e9e6d52704224b776efcee7e2b86"
server_key_iv: "899a76f13c62b480b7351966eeafb2e7"
server_key_version: "encryptionkey@0"
I20260618 22:39:07.844319 11343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "820b4c77e26a4d16a9991df5a8432bdf"
format_stamp: "Formatted at 2026-06-18 22:39:07 on dist-test-slave-lnj3"
server_key: "4948e9e6d52704224b776efcee7e2b86"
server_key_iv: "899a76f13c62b480b7351966eeafb2e7"
server_key_version: "encryptionkey@0"
I20260618 22:39:07.849596 11343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260618 22:39:07.852909 11357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:07.854187 11343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260618 22:39:07.854337 11343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "820b4c77e26a4d16a9991df5a8432bdf"
format_stamp: "Formatted at 2026-06-18 22:39:07 on dist-test-slave-lnj3"
server_key: "4948e9e6d52704224b776efcee7e2b86"
server_key_iv: "899a76f13c62b480b7351966eeafb2e7"
server_key_version: "encryptionkey@0"
I20260618 22:39:07.854532 11343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:07.870970 11343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:07.871757 11343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:07.872030 11343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:07.888468 11343 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:39475
I20260618 22:39:07.888491 11408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:39475 every 8 connection(s)
I20260618 22:39:07.889743 11343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:39:07.891963 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 11343
I20260618 22:39:07.892151 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:39:07.892757 5975 external_mini_cluster.cc:1428] Setting key 6362c3ccff0d2e08615d44d6c45401ac
I20260618 22:39:07.893445 11409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:07.907423 11409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf: Bootstrap starting.
I20260618 22:39:07.907476 11412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:07.894560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.5.213.254:39475)
0618 22:39:07.895628 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:07.895641 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:07.895690 (+ 49us) server_negotiation.cc:407] Connection header received
0618 22:39:07.897912 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:07.897938 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:07.898526 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:07.899002 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:07.899519 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:07.901453 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:07.902688 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:07.903676 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:07.903861 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:07.904865 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:07.904875 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:07.904894 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:07.905582 (+ 688us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:07.905672 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:07.905700 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:07.905800 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:07.905963 (+ 163us) server_negotiation.cc:299] Negotiation successful
0618 22:39:07.906261 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":317,"threads_started":1}
I20260618 22:39:07.910570 11409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:07.910995 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:39:07.911602 11409 log.cc:824] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:07.914422 11409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf: No bootstrap required, opened a new log
I20260618 22:39:07.922070 11409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820b4c77e26a4d16a9991df5a8432bdf" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 39475 } }
I20260618 22:39:07.922425 11409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:07.922508 11409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820b4c77e26a4d16a9991df5a8432bdf, State: Initialized, Role: FOLLOWER
I20260618 22:39:07.923063 11409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820b4c77e26a4d16a9991df5a8432bdf" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 39475 } }
I20260618 22:39:07.923300 11409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:39:07.923403 11409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:39:07.923520 11409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:07.925489 11409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820b4c77e26a4d16a9991df5a8432bdf" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 39475 } }
I20260618 22:39:07.925928 11409 leader_election.cc:302] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 820b4c77e26a4d16a9991df5a8432bdf; no voters:
I20260618 22:39:07.926419 11409 leader_election.cc:288] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:39:07.926558 11414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:07.927564 11414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf [term 1 LEADER]: Becoming Leader. State: Replica: 820b4c77e26a4d16a9991df5a8432bdf, State: Running, Role: LEADER
I20260618 22:39:07.928063 11414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820b4c77e26a4d16a9991df5a8432bdf" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 39475 } }
I20260618 22:39:07.928457 11409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:39:07.930759 11415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "820b4c77e26a4d16a9991df5a8432bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820b4c77e26a4d16a9991df5a8432bdf" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 39475 } } }
I20260618 22:39:07.930807 11416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 820b4c77e26a4d16a9991df5a8432bdf. Latest consensus state: current_term: 1 leader_uuid: "820b4c77e26a4d16a9991df5a8432bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820b4c77e26a4d16a9991df5a8432bdf" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 39475 } } }
I20260618 22:39:07.931133 11415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:07.931162 11416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:07.932983 11420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:39:07.939427 11420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:39:07.948665 11420 catalog_manager.cc:1347] Generated new cluster ID: 8897007e0812404f8179f8f7ea8c9dc4
I20260618 22:39:07.948788 11420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:39:07.968109 11420 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:39:07.969348 11420 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:39:07.978654 11420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 820b4c77e26a4d16a9991df5a8432bdf: Generated new TSK 0
I20260618 22:39:07.979471 11420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:39:08.005507 11412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:07.998103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.5.213.254:39475)
0618 22:39:07.998328 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:07.998335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:07.999192 (+ 857us) server_negotiation.cc:407] Connection header received
0618 22:39:07.999410 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:07.999414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:07.999479 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:07.999610 (+ 131us) server_negotiation.cc:226] Negotiated authn=JWT
0618 22:39:08.000072 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:08.000810 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:08.002154 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:08.002569 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:08.002712 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:08.003200 (+ 488us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0618 22:39:08.004930 (+ 1730us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35073/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0618 22:39:08.005228 (+ 298us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39868: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35073/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":80}
W20260618 22:39:08.005771 11438 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:07.998006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:39475 (local address 127.0.0.1:39868)
0618 22:39:07.999076 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:07.999094 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:07.999237 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:07.999677 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:07.999683 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:07.999696 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0618 22:39:07.999931 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:07.999940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:08.000946 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:08.000950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:08.002005 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:08.002016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:08.002769 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:08.002777 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:08.002914 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:08.002946 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0618 22:39:08.005104 (+ 2158us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35073/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0618 22:39:08.005358 (+ 254us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.213.254:39475: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35073/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":942,"thread_start_us":803,"threads_started":1}
W20260618 22:39:08.005900 11412 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39868: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35073/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260618 22:39:08.006551 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 11343
I20260618 22:39:08.007607 11404 generic_service.cc:196] Checking for leaks (request via RPC)
E20260618 22:39:08.017472 11439 webserver.cc:533] Webserver: sslize failed:
E20260618 22:39:08.017578 11439 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260618 22:39:08.095338 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 11343
2026-06-18T22:39:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (684 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-18T22:39:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:39:08Z Disabled control of system clock
I20260618 22:39:08.168632 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:34615
--webserver_interface=127.5.213.254
--webserver_port=0
--builtin_ntp_servers=127.5.213.212:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:34615
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781822307836422-5975-0/testchainca.cert with env {}
W20260618 22:39:08.472723 11444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:08.473024 11444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:08.473129 11444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:08.480111 11444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:39:08.480217 11444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:08.480261 11444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:08.480314 11444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:39:08.480363 11444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260618 22:39:08.480422 11444 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260618 22:39:08.480458 11444 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260618 22:39:08.489997 11444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:34615
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:34615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781822307836422-5975-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:08.491240 11444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:08.492442 11444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:08.503216 11452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:08.503280 11449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:08.503284 11450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:08.504101 11444 server_base.cc:1034] running on GCE node
I20260618 22:39:08.504778 11444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:08.506094 11444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:08.507326 11444 hybrid_clock.cc:648] HybridClock initialized: now 1781822348507258 us; error 54 us; skew 500 ppm
I20260618 22:39:08.507730 11444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:39:08.509965 11444 webserver.cc:460] Webserver started at http://127.5.213.254:35593/ using document root <none> and password file <none>
I20260618 22:39:08.510696 11444 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:08.510834 11444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:08.511157 11444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:08.514091 11444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "4161a8f2e6044a65a8d96ac7717bfce4"
format_stamp: "Formatted at 2026-06-18 22:39:08 on dist-test-slave-lnj3"
server_key: "4313a2ed53dde9c6507466aac691f4f3"
server_key_iv: "814860b026b100b05ce40de0989df4d5"
server_key_version: "encryptionkey@0"
I20260618 22:39:08.514954 11444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "4161a8f2e6044a65a8d96ac7717bfce4"
format_stamp: "Formatted at 2026-06-18 22:39:08 on dist-test-slave-lnj3"
server_key: "4313a2ed53dde9c6507466aac691f4f3"
server_key_iv: "814860b026b100b05ce40de0989df4d5"
server_key_version: "encryptionkey@0"
I20260618 22:39:08.520437 11444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:39:08.524241 11458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:08.525723 11444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:39:08.525919 11444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "4161a8f2e6044a65a8d96ac7717bfce4"
format_stamp: "Formatted at 2026-06-18 22:39:08 on dist-test-slave-lnj3"
server_key: "4313a2ed53dde9c6507466aac691f4f3"
server_key_iv: "814860b026b100b05ce40de0989df4d5"
server_key_version: "encryptionkey@0"
I20260618 22:39:08.526135 11444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:08.543473 11444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:08.544333 11444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:08.544595 11444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:08.560937 11444 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:34615
I20260618 22:39:08.560945 11509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:34615 every 8 connection(s)
I20260618 22:39:08.562217 11444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:39:08.564456 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 11444
I20260618 22:39:08.564663 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:39:08.565472 5975 external_mini_cluster.cc:1428] Setting key 693988c779f7c3ec7a5e4c80ecbbded9
I20260618 22:39:08.566754 11510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:08.579245 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:08.567512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33544 (local address 127.5.213.254:34615)
0618 22:39:08.568431 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:08.568442 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:08.568481 (+ 39us) server_negotiation.cc:407] Connection header received
0618 22:39:08.569987 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:08.570009 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:08.570492 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:08.570899 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:08.571506 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:08.572931 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:08.574747 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:08.575398 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:08.575539 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:08.576738 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:08.576748 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:08.576762 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:08.577329 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:08.577399 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:08.577437 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:08.577530 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:08.577746 (+ 216us) server_negotiation.cc:299] Negotiation successful
0618 22:39:08.578080 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":390,"threads_started":1}
I20260618 22:39:08.579932 11510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4: Bootstrap starting.
I20260618 22:39:08.582862 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:39:08.583135 11510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:08.584061 11510 log.cc:824] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:08.586853 11510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4: No bootstrap required, opened a new log
I20260618 22:39:08.593999 11510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4161a8f2e6044a65a8d96ac7717bfce4" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 34615 } }
I20260618 22:39:08.594374 11510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:08.594475 11510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4161a8f2e6044a65a8d96ac7717bfce4, State: Initialized, Role: FOLLOWER
I20260618 22:39:08.595058 11510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4161a8f2e6044a65a8d96ac7717bfce4" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 34615 } }
I20260618 22:39:08.595283 11510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:39:08.595398 11510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:39:08.595557 11510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:08.597584 11510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4161a8f2e6044a65a8d96ac7717bfce4" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 34615 } }
I20260618 22:39:08.598050 11510 leader_election.cc:302] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4161a8f2e6044a65a8d96ac7717bfce4; no voters:
I20260618 22:39:08.598603 11510 leader_election.cc:288] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:39:08.598789 11515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:08.599392 11515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4 [term 1 LEADER]: Becoming Leader. State: Replica: 4161a8f2e6044a65a8d96ac7717bfce4, State: Running, Role: LEADER
I20260618 22:39:08.599948 11515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4161a8f2e6044a65a8d96ac7717bfce4" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 34615 } }
I20260618 22:39:08.600733 11510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:39:08.602453 11516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4161a8f2e6044a65a8d96ac7717bfce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4161a8f2e6044a65a8d96ac7717bfce4" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 34615 } } }
I20260618 22:39:08.602479 11517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4161a8f2e6044a65a8d96ac7717bfce4. Latest consensus state: current_term: 1 leader_uuid: "4161a8f2e6044a65a8d96ac7717bfce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4161a8f2e6044a65a8d96ac7717bfce4" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 34615 } } }
I20260618 22:39:08.602875 11517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:08.602872 11516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:08.605119 11520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:39:08.610133 11520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:39:08.619081 11520 catalog_manager.cc:1347] Generated new cluster ID: d682ee1e1f4e4176b09b5e1e16de76b0
I20260618 22:39:08.619197 11520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:39:08.635779 11520 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:39:08.637068 11520 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:39:08.648092 11520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4161a8f2e6044a65a8d96ac7717bfce4: Generated new TSK 0
I20260618 22:39:08.648898 11520 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:39:08.747859 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:08.665938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33548 (local address 127.5.213.254:34615)
0618 22:39:08.666121 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:08.666127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:08.666428 (+ 301us) server_negotiation.cc:407] Connection header received
0618 22:39:08.666609 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:08.666615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:08.666720 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:08.666863 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0618 22:39:08.667262 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:08.668370 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:08.669685 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:08.670368 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:08.670483 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:08.670983 (+ 500us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0618 22:39:08.747169 (+ 76186us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0618 22:39:08.747558 (+ 389us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33548: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":64}
W20260618 22:39:08.747993 11513 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33548: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260618 22:39:08.748034 11539 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:08.665822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:34615 (local address 127.0.0.1:33548)
0618 22:39:08.666339 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:08.666354 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:08.666463 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:08.666871 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:08.666874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:08.666884 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0618 22:39:08.667129 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:08.667137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:08.668501 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:08.668505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:08.669559 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:08.669571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:08.670523 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:08.670528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:08.670696 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:08.670728 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0618 22:39:08.747470 (+ 76742us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0618 22:39:08.747709 (+ 239us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.213.254:34615: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":269,"threads_started":1}
I20260618 22:39:08.749913 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 11444
I20260618 22:39:08.751026 11505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:08.847747 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 11444
2026-06-18T22:39:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (732 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (386 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (387 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:39:09 dist-test-slave-lnj3 krb5kdc[11554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:39:09 dist-test-slave-lnj3 krb5kdc[11554](info): set up 2 sockets
Jun 18 22:39:09 dist-test-slave-lnj3 krb5kdc[11554](info): commencing operation
krb5kdc: starting...
W20260618 22:39:10.615024 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:39:10 dist-test-slave-lnj3 krb5kdc[11554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:39:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:39:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:39:10.800164 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:37043
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:37043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:11.106524 11567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:11.106877 11567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:11.106995 11567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:11.114259 11567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:39:11.114373 11567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:11.114430 11567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:11.114511 11567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:39:11.114570 11567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:39:11.124522 11567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:37043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:37043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:11.125877 11567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:11.127190 11567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:11.138166 11572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:11.138325 11575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:11.138350 11573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:11.139367 11567 server_base.cc:1034] running on GCE node
I20260618 22:39:11.140067 11567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:11.141429 11567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:11.142661 11567 hybrid_clock.cc:648] HybridClock initialized: now 1781822351142587 us; error 36 us; skew 500 ppm
I20260618 22:39:11.142983 11567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:11 dist-test-slave-lnj3 krb5kdc[11554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822351, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:11.145849 11567 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:39:11.147303 11567 webserver.cc:460] Webserver started at http://127.5.213.254:42039/ using document root <none> and password file <none>
I20260618 22:39:11.148007 11567 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:11.148128 11567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:11.148409 11567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:11.151341 11567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "6aa193b93de14d1ba3d945a3887605b8"
format_stamp: "Formatted at 2026-06-18 22:39:11 on dist-test-slave-lnj3"
server_key: "e76ccff388c0e57b9dce7e48aa825ea5"
server_key_iv: "6ac4838096ac1c8655822ebccc217555"
server_key_version: "encryptionkey@0"
I20260618 22:39:11.152098 11567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "6aa193b93de14d1ba3d945a3887605b8"
format_stamp: "Formatted at 2026-06-18 22:39:11 on dist-test-slave-lnj3"
server_key: "e76ccff388c0e57b9dce7e48aa825ea5"
server_key_iv: "6ac4838096ac1c8655822ebccc217555"
server_key_version: "encryptionkey@0"
I20260618 22:39:11.157312 11567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:39:11.160893 11582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:11.162220 11567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:39:11.162381 11567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "6aa193b93de14d1ba3d945a3887605b8"
format_stamp: "Formatted at 2026-06-18 22:39:11 on dist-test-slave-lnj3"
server_key: "e76ccff388c0e57b9dce7e48aa825ea5"
server_key_iv: "6ac4838096ac1c8655822ebccc217555"
server_key_version: "encryptionkey@0"
I20260618 22:39:11.162556 11567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:11.178398 11567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:11.182436 11567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:11.182716 11567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:11.199136 11567 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:37043
I20260618 22:39:11.199162 11633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:37043 every 8 connection(s)
I20260618 22:39:11.200434 11567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:39:11.203975 11634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:11.207351 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 11567
I20260618 22:39:11.207523 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:39:11.208091 5975 external_mini_cluster.cc:1428] Setting key cd46e5d9a2eacf51b7e4546280a8748f
I20260618 22:39:11.214787 11634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8: Bootstrap starting.
I20260618 22:39:11.218760 11634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:11.219946 11634 log.cc:824] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:11.223495 11634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8: No bootstrap required, opened a new log
Jun 18 22:39:11 dist-test-slave-lnj3 krb5kdc[11554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:39:11.232043 11637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:11.210002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33102 (local address 127.5.213.254:37043)
0618 22:39:11.210788 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:11.210800 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:11.210838 (+ 38us) server_negotiation.cc:407] Connection header received
0618 22:39:11.213129 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:11.213155 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:11.213552 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:11.213886 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:11.215877 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:11.217350 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:11.219208 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:11.219932 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:11.220093 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:11.224922 (+ 4829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:11.224931 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:11.224950 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:11.224989 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:11.228086 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:11.228771 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:11.228778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:11.228785 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:11.228950 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:11.229315 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:11.229319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:11.229321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:11.229878 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:11.230174 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:11.230615 (+ 441us) server_negotiation.cc:299] Negotiation successful
0618 22:39:11.231046 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":341,"threads_started":1}
I20260618 22:39:11.231817 11634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa193b93de14d1ba3d945a3887605b8" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 37043 } }
I20260618 22:39:11.232197 11634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:11.232311 11634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aa193b93de14d1ba3d945a3887605b8, State: Initialized, Role: FOLLOWER
I20260618 22:39:11.232898 11634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa193b93de14d1ba3d945a3887605b8" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 37043 } }
I20260618 22:39:11.233152 11634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:39:11.233302 11634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:39:11.233497 11634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:11.236001 11634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa193b93de14d1ba3d945a3887605b8" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 37043 } }
I20260618 22:39:11.236510 11634 leader_election.cc:302] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6aa193b93de14d1ba3d945a3887605b8; no voters:
I20260618 22:39:11.237073 11634 leader_election.cc:288] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:39:11.237179 11639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:11.237867 11639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8 [term 1 LEADER]: Becoming Leader. State: Replica: 6aa193b93de14d1ba3d945a3887605b8, State: Running, Role: LEADER
I20260618 22:39:11.238386 11639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa193b93de14d1ba3d945a3887605b8" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 37043 } }
I20260618 22:39:11.239189 11634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:39:11.240860 11641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6aa193b93de14d1ba3d945a3887605b8. Latest consensus state: current_term: 1 leader_uuid: "6aa193b93de14d1ba3d945a3887605b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa193b93de14d1ba3d945a3887605b8" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 37043 } } }
I20260618 22:39:11.241336 11641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:11.241514 11640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6aa193b93de14d1ba3d945a3887605b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa193b93de14d1ba3d945a3887605b8" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 37043 } } }
I20260618 22:39:11.241888 11640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:11.243227 11644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:39:11.248157 11644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:39:11.258457 11644 catalog_manager.cc:1347] Generated new cluster ID: a99afcb2d85a4b66b8bb8e32e3944d18
I20260618 22:39:11.258652 11644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:39:11.296295 11644 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:39:11.297598 11644 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:39:11.304833 11644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6aa193b93de14d1ba3d945a3887605b8: Generated new TSK 0
I20260618 22:39:11.305644 11644 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
I20260618 22:39:11.362161 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:37043
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:11.714103 11660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:11.714357 11660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:11.714445 11660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:11.721617 11660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:11.721724 11660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:11.721838 11660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:39:11.731446 11660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:37043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:11.732638 11660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:11.733968 11660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:11.746107 11665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:11.746266 11668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:11.746289 11666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:11.747304 11660 server_base.cc:1034] running on GCE node
I20260618 22:39:11.747884 11660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:11.748797 11660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:11.749990 11660 hybrid_clock.cc:648] HybridClock initialized: now 1781822351749942 us; error 38 us; skew 500 ppm
I20260618 22:39:11.750296 11660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:11 dist-test-slave-lnj3 krb5kdc[11554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822351, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:11.753264 11660 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:39:11.754585 11660 webserver.cc:460] Webserver started at http://127.5.213.193:41995/ using document root <none> and password file <none>
I20260618 22:39:11.755247 11660 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:11.755355 11660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:11.755623 11660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:11.758386 11660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "d4f6304eb3f2439a97a029772e2b2911"
format_stamp: "Formatted at 2026-06-18 22:39:11 on dist-test-slave-lnj3"
server_key: "023925545df3ccd508928eb855daadf4"
server_key_iv: "fe46ed0d780d206d27bdc88edfb9dd87"
server_key_version: "encryptionkey@0"
I20260618 22:39:11.759184 11660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "d4f6304eb3f2439a97a029772e2b2911"
format_stamp: "Formatted at 2026-06-18 22:39:11 on dist-test-slave-lnj3"
server_key: "023925545df3ccd508928eb855daadf4"
server_key_iv: "fe46ed0d780d206d27bdc88edfb9dd87"
server_key_version: "encryptionkey@0"
I20260618 22:39:11.764178 11660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260618 22:39:11.767536 11675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:11.768793 11660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260618 22:39:11.768937 11660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "d4f6304eb3f2439a97a029772e2b2911"
format_stamp: "Formatted at 2026-06-18 22:39:11 on dist-test-slave-lnj3"
server_key: "023925545df3ccd508928eb855daadf4"
server_key_iv: "fe46ed0d780d206d27bdc88edfb9dd87"
server_key_version: "encryptionkey@0"
I20260618 22:39:11.769117 11660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:11.785696 11660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:11.789184 11660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:11.789425 11660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:11.790256 11660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:11.791508 11660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:11.791581 11660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:11.791698 11660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:11.791749 11660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:11.820101 11660 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:43415
I20260618 22:39:11.820134 11787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:43415 every 8 connection(s)
I20260618 22:39:11.821278 11660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:39:11.822928 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 11660
I20260618 22:39:11.823113 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:39:11.823701 5975 external_mini_cluster.cc:1428] Setting key 28130f7e77d9e6ff22b8a4927ff087de
Jun 18 22:39:11 dist-test-slave-lnj3 krb5kdc[11554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822351, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.213.194@KRBTEST.COM; defaulting to no policy
I20260618 22:39:11.839900 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:11.823801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:52937 (local address 127.5.213.254:37043)
0618 22:39:11.824290 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:11.824297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:11.825685 (+ 1388us) server_negotiation.cc:407] Connection header received
0618 22:39:11.827196 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:11.827201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:11.827284 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:11.827374 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:11.828816 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:11.829496 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:11.831052 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:11.831436 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:11.831544 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:11.834694 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:11.834698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:11.834702 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:11.834732 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:11.837040 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:11.837705 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:11.837711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:11.837713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:11.837833 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:11.838221 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:11.838225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:11.838229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:11.838483 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:11.838620 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:11.839289 (+ 669us) server_negotiation.cc:299] Negotiation successful
0618 22:39:11.839499 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":267,"threads_started":1}
I20260618 22:39:11.840873 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:11.824423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:37043 (local address 127.5.213.193:52937)
0618 22:39:11.825253 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:11.825328 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:11.826956 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:11.827611 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:11.827618 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:11.828096 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:11.828651 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:11.828668 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:11.829645 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:11.829649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:11.830925 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:11.830936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:11.831645 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:11.831652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:11.831870 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:11.832460 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:11.832485 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:11.834475 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:11.837164 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:11.837173 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:11.837190 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:11.837593 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:11.837933 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:11.837938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:11.837943 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:11.838119 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:11.838600 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:11.838608 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:11.838954 (+ 346us) client_negotiation.cc:769] Sending connection context
0618 22:39:11.839187 (+ 233us) client_negotiation.cc:240] Negotiation successful
0618 22:39:11.839690 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":332,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.194@KRBTEST.COM" created.
I20260618 22:39:11.842267 11788 heartbeater.cc:347] Connected to a master server at 127.5.213.254:37043
I20260618 22:39:11.842610 11788 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:11.843487 11788 heartbeater.cc:511] Master 127.5.213.254:37043 requested a full tablet report, sending...
I20260618 22:39:11.845638 11599 ts_manager.cc:194] Registered new tserver with Master: d4f6304eb3f2439a97a029772e2b2911 (127.5.213.193:43415)
I20260618 22:39:11.847054 11599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:52937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
I20260618 22:39:11.863637 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:37043
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:12.168089 11794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:12.168342 11794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:12.168440 11794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:12.175525 11794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:12.175639 11794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:12.175757 11794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:39:12.185201 11794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:37043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:12.186367 11794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:12.187585 11794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:12.199335 11799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:12.199359 11800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:12.199560 11802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:12.200500 11794 server_base.cc:1034] running on GCE node
I20260618 22:39:12.201040 11794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:12.201881 11794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:12.203064 11794 hybrid_clock.cc:648] HybridClock initialized: now 1781822352203000 us; error 59 us; skew 500 ppm
I20260618 22:39:12.203343 11794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:12 dist-test-slave-lnj3 krb5kdc[11554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822352, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:12.206179 11794 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:39:12.207598 11794 webserver.cc:460] Webserver started at http://127.5.213.194:39345/ using document root <none> and password file <none>
I20260618 22:39:12.208223 11794 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:12.208338 11794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:12.208622 11794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:12.211515 11794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "6063598552b547cc9c22a2b6d5b0ddf9"
format_stamp: "Formatted at 2026-06-18 22:39:12 on dist-test-slave-lnj3"
server_key: "dc139aaf40c464071f1b7c9fd9201850"
server_key_iv: "f86f0f46ffbcf4b88bac6e997c4f80b7"
server_key_version: "encryptionkey@0"
I20260618 22:39:12.212222 11794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "6063598552b547cc9c22a2b6d5b0ddf9"
format_stamp: "Formatted at 2026-06-18 22:39:12 on dist-test-slave-lnj3"
server_key: "dc139aaf40c464071f1b7c9fd9201850"
server_key_iv: "f86f0f46ffbcf4b88bac6e997c4f80b7"
server_key_version: "encryptionkey@0"
I20260618 22:39:12.217070 11794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:39:12.220429 11809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:12.221781 11794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260618 22:39:12.221912 11794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "6063598552b547cc9c22a2b6d5b0ddf9"
format_stamp: "Formatted at 2026-06-18 22:39:12 on dist-test-slave-lnj3"
server_key: "dc139aaf40c464071f1b7c9fd9201850"
server_key_iv: "f86f0f46ffbcf4b88bac6e997c4f80b7"
server_key_version: "encryptionkey@0"
I20260618 22:39:12.222066 11794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:12.258107 11794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:12.262279 11794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:12.262538 11794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:12.263471 11794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:12.264750 11794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:12.264819 11794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:12.264938 11794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:12.264986 11794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:12.292757 11794 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:33553
I20260618 22:39:12.292825 11921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:33553 every 8 connection(s)
I20260618 22:39:12.293988 11794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:39:12.295135 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 11794
I20260618 22:39:12.295295 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:39:12.295830 5975 external_mini_cluster.cc:1428] Setting key f639b0856aee4e2d353156b5f30a327a
Jun 18 22:39:12 dist-test-slave-lnj3 krb5kdc[11554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822352, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.213.195@KRBTEST.COM; defaulting to no policy
I20260618 22:39:12.315384 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:12.296956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:54803 (local address 127.5.213.254:37043)
0618 22:39:12.297152 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:12.297159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:12.298440 (+ 1281us) server_negotiation.cc:407] Connection header received
0618 22:39:12.300824 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:12.300828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:12.300896 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:12.300982 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:12.303044 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:12.303762 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:12.305607 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:12.306261 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:12.306382 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:12.309871 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:12.309876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:12.309880 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:12.309912 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:12.312295 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:12.313150 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:12.313154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:12.313156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:12.313278 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:12.313707 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:12.313710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:12.313712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:12.313958 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:12.314066 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:12.314930 (+ 864us) server_negotiation.cc:299] Negotiation successful
0618 22:39:12.315093 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260618 22:39:12.316529 11924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:12.297416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:37043 (local address 127.5.213.194:54803)
0618 22:39:12.298257 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:12.298324 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:12.300518 (+ 2194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:12.301326 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:12.301336 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:12.302057 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:12.302871 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:12.302897 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:12.303903 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:12.303908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:12.305470 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:12.305481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:12.306471 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:12.306476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:12.306724 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:12.307336 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:12.307363 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:12.309623 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:12.312430 (+ 2807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:12.312440 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:12.312459 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:12.313020 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:12.313372 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:12.313377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:12.313381 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:12.313615 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:12.314074 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:12.314082 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:12.314520 (+ 438us) client_negotiation.cc:769] Sending connection context
0618 22:39:12.314859 (+ 339us) client_negotiation.cc:240] Negotiation successful
0618 22:39:12.315396 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":361,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.195@KRBTEST.COM" created.
I20260618 22:39:12.317864 11922 heartbeater.cc:347] Connected to a master server at 127.5.213.254:37043
I20260618 22:39:12.318142 11922 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:12.318866 11922 heartbeater.cc:511] Master 127.5.213.254:37043 requested a full tablet report, sending...
I20260618 22:39:12.320201 11599 ts_manager.cc:194] Registered new tserver with Master: 6063598552b547cc9c22a2b6d5b0ddf9 (127.5.213.194:33553)
I20260618 22:39:12.320961 11599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:54803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
I20260618 22:39:12.339243 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:37043
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:12.647780 11927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:12.648042 11927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:12.648172 11927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:12.655166 11927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:12.655260 11927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:12.655403 11927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:39:12.664741 11927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:37043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:12.666013 11927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:12.667338 11927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:12.679791 11935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:12.679919 11932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:12.679973 11933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:12.680773 11927 server_base.cc:1034] running on GCE node
I20260618 22:39:12.681445 11927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:12.682448 11927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:12.683748 11927 hybrid_clock.cc:648] HybridClock initialized: now 1781822352683665 us; error 68 us; skew 500 ppm
I20260618 22:39:12.684067 11927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:12 dist-test-slave-lnj3 krb5kdc[11554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822352, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:12.687333 11927 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:39:12.689013 11927 webserver.cc:460] Webserver started at http://127.5.213.195:39469/ using document root <none> and password file <none>
I20260618 22:39:12.689749 11927 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:12.689874 11927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:12.690161 11927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:12.693143 11927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "0ee811b45be14c4e85df2f2d1a9fd483"
format_stamp: "Formatted at 2026-06-18 22:39:12 on dist-test-slave-lnj3"
server_key: "fb2eb23b5c0da26215941a6cccd815be"
server_key_iv: "4825c9d8ea5c58b0044fdb4975c21133"
server_key_version: "encryptionkey@0"
I20260618 22:39:12.693894 11927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "0ee811b45be14c4e85df2f2d1a9fd483"
format_stamp: "Formatted at 2026-06-18 22:39:12 on dist-test-slave-lnj3"
server_key: "fb2eb23b5c0da26215941a6cccd815be"
server_key_iv: "4825c9d8ea5c58b0044fdb4975c21133"
server_key_version: "encryptionkey@0"
I20260618 22:39:12.698984 11927 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260618 22:39:12.702279 11942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:12.703634 11927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:39:12.703778 11927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "0ee811b45be14c4e85df2f2d1a9fd483"
format_stamp: "Formatted at 2026-06-18 22:39:12 on dist-test-slave-lnj3"
server_key: "fb2eb23b5c0da26215941a6cccd815be"
server_key_iv: "4825c9d8ea5c58b0044fdb4975c21133"
server_key_version: "encryptionkey@0"
I20260618 22:39:12.703974 11927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:12.723194 11927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:12.727764 11927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:12.728044 11927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:12.729025 11927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:12.730449 11927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:12.730525 11927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:12.730671 11927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:12.730726 11927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260618 22:39:12.761206 11927 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:44217
I20260618 22:39:12.761242 12054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:44217 every 8 connection(s)
I20260618 22:39:12.762521 11927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:39:12.767211 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 11927
I20260618 22:39:12.767395 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:39:12.767840 5975 external_mini_cluster.cc:1428] Setting key d1049811762788483fbe3046e6f23f94
Jun 18 22:39:12 dist-test-slave-lnj3 krb5kdc[11554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822352, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:39:12.782377 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:12.765291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:50093 (local address 127.5.213.254:37043)
0618 22:39:12.765505 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:12.765510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:12.766840 (+ 1330us) server_negotiation.cc:407] Connection header received
0618 22:39:12.768811 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:12.768816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:12.768924 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:12.769008 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:12.770548 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:12.771400 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:12.773094 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:12.773567 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:12.773682 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:12.777064 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:12.777069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:12.777072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:12.777112 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:12.779411 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:12.780086 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:12.780091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:12.780103 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:12.780251 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:12.780649 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:12.780654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:12.780657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:12.780915 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:12.781037 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:12.781821 (+ 784us) server_negotiation.cc:299] Negotiation successful
0618 22:39:12.782031 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260618 22:39:12.783463 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:12.765768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:37043 (local address 127.5.213.195:50093)
0618 22:39:12.766662 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:12.766719 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:12.768502 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:12.769270 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:12.769280 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:12.769811 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:12.770387 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:12.770404 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:12.771550 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:12.771554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:12.772920 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:12.772932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:12.773794 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:12.773801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:12.774034 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:12.774713 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:12.774741 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:12.776858 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:12.779541 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:12.779551 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:12.779568 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:12.779973 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:12.780353 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:12.780358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:12.780361 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:12.780549 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:12.781031 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:12.781039 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:12.781381 (+ 342us) client_negotiation.cc:769] Sending connection context
0618 22:39:12.781693 (+ 312us) client_negotiation.cc:240] Negotiation successful
0618 22:39:12.782220 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":353,"threads_started":1}
I20260618 22:39:12.784778 12055 heartbeater.cc:347] Connected to a master server at 127.5.213.254:37043
I20260618 22:39:12.785064 12055 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:12.785781 12055 heartbeater.cc:511] Master 127.5.213.254:37043 requested a full tablet report, sending...
I20260618 22:39:12.787184 11599 ts_manager.cc:194] Registered new tserver with Master: 0ee811b45be14c4e85df2f2d1a9fd483 (127.5.213.195:44217)
I20260618 22:39:12.787977 11599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:50093
I20260618 22:39:12.795462 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/test-admin.keytab.
Jun 18 22:39:12 dist-test-slave-lnj3 krb5kdc[11554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:12.820509 5975 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260618 22:39:12.826139 12062 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:12.823254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39466 (local address 127.5.213.193:43415)
0618 22:39:12.824226 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:12.824236 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:12.824264 (+ 28us) server_negotiation.cc:407] Connection header received
0618 22:39:12.824325 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:12.824332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:12.824574 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:12.824740 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:12.825749 (+ 1009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39466: BlockingRecv error: recv got EOF from 127.0.0.1:39466 (error 108)
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":660,"threads_started":1}
W20260618 22:39:12.826486 5975 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/krb5cc' of type FILE
Jun 18 22:39:12 dist-test-slave-lnj3 krb5kdc[11554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:12.828279 5975 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 18 22:39:12 dist-test-slave-lnj3 krb5kdc[11554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:12.829579 5975 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 18 22:39:12 dist-test-slave-lnj3 krb5kdc[11554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:39:12.847471 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:12.831174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39480 (local address 127.5.213.193:43415)
0618 22:39:12.831328 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:12.831332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:12.831642 (+ 310us) server_negotiation.cc:407] Connection header received
0618 22:39:12.831795 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:12.831799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:12.831859 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:12.831944 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:12.832940 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:12.833920 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:12.835151 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:12.835720 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:12.835811 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:12.839181 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:12.839192 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:12.839207 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:12.839257 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:12.844918 (+ 5661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:12.845560 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:12.845569 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:12.845573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:12.845695 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:12.845978 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:12.845980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:12.845982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:12.846335 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:12.846529 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:12.846954 (+ 425us) server_negotiation.cc:299] Negotiation successful
0618 22:39:12.847198 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 22:39:12.849718 11788 heartbeater.cc:503] Master 127.5.213.254:37043 was elected leader, sending a full tablet report...
W20260618 22:39:12.853185 12062 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:12.851604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39492 (local address 127.5.213.193:43415)
0618 22:39:12.851756 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:12.851760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:12.852102 (+ 342us) server_negotiation.cc:407] Connection header received
0618 22:39:12.852221 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:12.852224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:12.852288 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:12.852360 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:12.852988 (+ 628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39492: BlockingRecv error: recv got EOF from 127.0.0.1:39492 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260618 22:39:12.853381 5975 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/krb5cc' of type FILE
Jun 18 22:39:12 dist-test-slave-lnj3 krb5kdc[11554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:12.854916 5975 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 18 22:39:12 dist-test-slave-lnj3 krb5kdc[11554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:12.856068 5975 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 18 22:39:12 dist-test-slave-lnj3 krb5kdc[11554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:39:12.869706 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:12.857237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39502 (local address 127.5.213.193:43415)
0618 22:39:12.857379 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:12.857385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:12.857699 (+ 314us) server_negotiation.cc:407] Connection header received
0618 22:39:12.857836 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:12.857841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:12.857893 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:12.857965 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:12.858873 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:12.859531 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:12.860685 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:12.861093 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:12.861205 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:12.864711 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:12.864716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:12.864720 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:12.864756 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:12.867345 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:12.868076 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:12.868079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:12.868082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:12.868201 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:12.868541 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:12.868543 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:12.868545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:12.868811 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:12.868998 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:12.869258 (+ 260us) server_negotiation.cc:299] Negotiation successful
0618 22:39:12.869421 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260618 22:39:12.874125 12062 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:12.872328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39506 (local address 127.5.213.193:43415)
0618 22:39:12.872531 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:12.872537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:12.872762 (+ 225us) server_negotiation.cc:407] Connection header received
0618 22:39:12.872949 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:12.872954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:12.873030 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:12.873109 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:12.873885 (+ 776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39506: BlockingRecv error: recv got EOF from 127.0.0.1:39506 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260618 22:39:12.874257 5975 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781822307836422-5975-0/krb5kdc/krb5cc' of type FILE
Jun 18 22:39:12 dist-test-slave-lnj3 krb5kdc[11554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:12.875752 5975 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 18 22:39:12 dist-test-slave-lnj3 krb5kdc[11554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:12.876986 5975 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 18 22:39:12 dist-test-slave-lnj3 krb5kdc[11554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:39:12.889647 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:12.878287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39522 (local address 127.5.213.193:43415)
0618 22:39:12.878468 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:12.878474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:12.878596 (+ 122us) server_negotiation.cc:407] Connection header received
0618 22:39:12.878755 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:12.878758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:12.878813 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:12.878887 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:12.879700 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:12.880381 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:12.881519 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:12.881945 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:12.882039 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:12.885154 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:12.885159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:12.885164 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:12.885192 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:12.887512 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:12.888035 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:12.888038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:12.888040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:12.888191 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:12.888454 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:12.888457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:12.888459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:12.888703 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:12.888913 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:12.889151 (+ 238us) server_negotiation.cc:299] Negotiation successful
0618 22:39:12.889360 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 22:39:12.891196 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 11660
I20260618 22:39:12.901815 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:12.891827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39532 (local address 127.5.213.193:43415)
0618 22:39:12.891968 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:12.891971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:12.892300 (+ 329us) server_negotiation.cc:407] Connection header received
0618 22:39:12.892415 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:12.892418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:12.892470 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:12.892540 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:12.893367 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:12.894029 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:12.895234 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:12.895639 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:12.895727 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:12.897479 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:12.897483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:12.897486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:12.897513 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:12.899791 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:12.900335 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:12.900338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:12.900340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:12.900455 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:12.900791 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:12.900794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:12.900796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:12.901024 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:12.901197 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:12.901412 (+ 215us) server_negotiation.cc:299] Negotiation successful
0618 22:39:12.901559 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:39:12.902361 11783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:12.982951 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 11660
I20260618 22:39:12.995517 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 11794
Jun 18 22:39:13 dist-test-slave-lnj3 krb5kdc[11554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.194@KRBTEST.COM
I20260618 22:39:13.010088 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:12.996350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.5.213.194:33553)
0618 22:39:12.996903 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:12.996910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:12.996936 (+ 26us) server_negotiation.cc:407] Connection header received
0618 22:39:12.996998 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:12.997005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:12.997219 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:12.997352 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:12.998248 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:12.999369 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:13.000577 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:13.001123 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:13.001216 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:13.004859 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:13.004872 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:13.004888 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:13.004933 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:13.007557 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:13.008229 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:13.008239 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:13.008243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:13.008375 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:13.008709 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:13.008711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:13.008714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:13.009095 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:13.009308 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:13.009573 (+ 265us) server_negotiation.cc:299] Negotiation successful
0618 22:39:13.009778 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":290,"threads_started":1}
I20260618 22:39:13.011476 11917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:13.090873 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 11794
I20260618 22:39:13.103173 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 11927
Jun 18 22:39:13 dist-test-slave-lnj3 krb5kdc[11554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.195@KRBTEST.COM
I20260618 22:39:13.117486 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:13.104113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50436 (local address 127.5.213.195:44217)
0618 22:39:13.104764 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:13.104771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:13.104798 (+ 27us) server_negotiation.cc:407] Connection header received
0618 22:39:13.104854 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:13.104860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:13.105058 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:13.105206 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:13.106196 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:13.107267 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:13.108491 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:13.109031 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:13.109138 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:13.112762 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:13.112771 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:13.112785 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:13.112819 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:13.115243 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:13.115824 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:13.115830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:13.115832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:13.115939 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:13.116246 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:13.116249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:13.116251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:13.116574 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:13.116798 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:13.116994 (+ 196us) server_negotiation.cc:299] Negotiation successful
0618 22:39:13.117193 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":343,"threads_started":1}
I20260618 22:39:13.118826 12050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:13.199658 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 11927
I20260618 22:39:13.213194 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 11567
I20260618 22:39:13.214323 11629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:13.311714 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 11567
2026-06-18T22:39:13Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3694 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:39:13 dist-test-slave-lnj3 krb5kdc[12082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:39:13 dist-test-slave-lnj3 krb5kdc[12082](info): set up 2 sockets
Jun 18 22:39:13 dist-test-slave-lnj3 krb5kdc[12082](info): commencing operation
krb5kdc: starting...
W20260618 22:39:14.307017 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:39:14 dist-test-slave-lnj3 krb5kdc[12082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:39:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:39:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.254.keytab.
Entry for principal oryx/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.254.keytab.
I20260618 22:39:14.494249 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:33087
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.254.keytab
--principal=oryx/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:33087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:14.801509 12095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:14.801785 12095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:14.801908 12095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:14.808900 12095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:39:14.809003 12095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:14.809077 12095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:14.809142 12095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:39:14.809197 12095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:39:14.818699 12095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:33087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.254.keytab
--principal=oryx/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:33087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:14.819926 12095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:14.821135 12095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:14.831630 12100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:14.831686 12101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:14.831712 12103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:14.832808 12095 server_base.cc:1034] running on GCE node
I20260618 22:39:14.833492 12095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:14.834844 12095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:14.836037 12095 hybrid_clock.cc:648] HybridClock initialized: now 1781822354835983 us; error 48 us; skew 500 ppm
I20260618 22:39:14.836349 12095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:14 dist-test-slave-lnj3 krb5kdc[12082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822354, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:14.839308 12095 init.cc:377] Logged in from keytab as oryx/127.5.213.254@KRBTEST.COM (short username oryx)
I20260618 22:39:14.840636 12095 webserver.cc:460] Webserver started at http://127.5.213.254:43877/ using document root <none> and password file <none>
I20260618 22:39:14.841286 12095 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:14.841401 12095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:14.841668 12095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:14.844533 12095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "798ea03554f649e8b59eff0c77e07c60"
format_stamp: "Formatted at 2026-06-18 22:39:14 on dist-test-slave-lnj3"
server_key: "76ff9a28e31fe46ad16321ed3f555e22"
server_key_iv: "9668fee2f96b0e46d51a71643d473258"
server_key_version: "encryptionkey@0"
I20260618 22:39:14.845249 12095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "798ea03554f649e8b59eff0c77e07c60"
format_stamp: "Formatted at 2026-06-18 22:39:14 on dist-test-slave-lnj3"
server_key: "76ff9a28e31fe46ad16321ed3f555e22"
server_key_iv: "9668fee2f96b0e46d51a71643d473258"
server_key_version: "encryptionkey@0"
I20260618 22:39:14.850127 12095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260618 22:39:14.853507 12110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:14.854987 12095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260618 22:39:14.855149 12095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "798ea03554f649e8b59eff0c77e07c60"
format_stamp: "Formatted at 2026-06-18 22:39:14 on dist-test-slave-lnj3"
server_key: "76ff9a28e31fe46ad16321ed3f555e22"
server_key_iv: "9668fee2f96b0e46d51a71643d473258"
server_key_version: "encryptionkey@0"
I20260618 22:39:14.855335 12095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:14.879424 12095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:14.883780 12095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:14.884052 12095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:14.901075 12095 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:33087
I20260618 22:39:14.901117 12161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:33087 every 8 connection(s)
I20260618 22:39:14.902416 12095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:39:14.906219 12162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:14.911755 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 12095
I20260618 22:39:14.911953 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:39:14.912564 5975 external_mini_cluster.cc:1428] Setting key 5cd5b002c935ce40fb490bc7157f7408
I20260618 22:39:14.917304 12162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60: Bootstrap starting.
I20260618 22:39:14.921051 12162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:14.922209 12162 log.cc:824] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:14.925287 12162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60: No bootstrap required, opened a new log
Jun 18 22:39:14 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM
I20260618 22:39:14.933528 12162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } }
I20260618 22:39:14.933983 12162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:14.934085 12162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 798ea03554f649e8b59eff0c77e07c60, State: Initialized, Role: FOLLOWER
I20260618 22:39:14.934684 12162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } }
I20260618 22:39:14.934908 12162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:39:14.935035 12162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:39:14.935185 12162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:14.935497 12165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:14.914653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.5.213.254:33087)
0618 22:39:14.915480 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:14.915496 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:14.915544 (+ 48us) server_negotiation.cc:407] Connection header received
0618 22:39:14.917759 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:14.917787 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:14.918304 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:14.918732 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:14.919910 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:14.921809 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:14.923271 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:14.924178 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:14.924343 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:14.928324 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:14.928332 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:14.928349 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:14.928384 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:14.931400 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:14.932153 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:14.932158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:14.932165 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:14.932341 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:14.932707 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:14.932710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:14.932712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:14.933189 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:14.933417 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:14.933837 (+ 420us) server_negotiation.cc:299] Negotiation successful
0618 22:39:14.934297 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":348,"threads_started":1}
I20260618 22:39:14.937389 12162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } }
I20260618 22:39:14.937852 12162 leader_election.cc:302] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 798ea03554f649e8b59eff0c77e07c60; no voters:
I20260618 22:39:14.938422 12162 leader_election.cc:288] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:39:14.938513 12167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:14.939143 12167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [term 1 LEADER]: Becoming Leader. State: Replica: 798ea03554f649e8b59eff0c77e07c60, State: Running, Role: LEADER
I20260618 22:39:14.939697 12167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } }
I20260618 22:39:14.940389 12162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:39:14.942585 12168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } }
I20260618 22:39:14.942617 12169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 798ea03554f649e8b59eff0c77e07c60. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } }
I20260618 22:39:14.942984 12169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:14.942984 12168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:14.944759 12174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:39:14.950227 12174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:39:14.959714 12174 catalog_manager.cc:1347] Generated new cluster ID: 5d782181ba1a42da86921e7b22f586f2
I20260618 22:39:14.959830 12174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:39:14.968617 12174 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:39:14.970317 12174 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:39:14.984689 12174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60: Generated new TSK 0
I20260618 22:39:14.985715 12174 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.193.keytab.
Entry for principal oryx/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.193.keytab.
I20260618 22:39:15.036312 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:33087
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.193.keytab
--principal=oryx/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:15.346726 12188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:15.347106 12188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:15.347267 12188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:15.354453 12188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:15.354547 12188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:15.354717 12188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:39:15.364212 12188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.193.keytab
--principal=oryx/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:33087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:15.365407 12188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:15.366951 12188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:15.379652 12193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:15.379675 12196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:15.379793 12194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:15.380276 12188 server_base.cc:1034] running on GCE node
I20260618 22:39:15.380817 12188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:15.381747 12188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:15.382948 12188 hybrid_clock.cc:648] HybridClock initialized: now 1781822355382889 us; error 51 us; skew 500 ppm
I20260618 22:39:15.383244 12188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:15 dist-test-slave-lnj3 krb5kdc[12082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822355, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:15.386157 12188 init.cc:377] Logged in from keytab as oryx/127.5.213.193@KRBTEST.COM (short username oryx)
I20260618 22:39:15.387512 12188 webserver.cc:460] Webserver started at http://127.5.213.193:32841/ using document root <none> and password file <none>
I20260618 22:39:15.388273 12188 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:15.388379 12188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:15.388608 12188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:15.391364 12188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "ebff97d8cce5416a95db1cf751e0018b"
format_stamp: "Formatted at 2026-06-18 22:39:15 on dist-test-slave-lnj3"
server_key: "973d91ce162a5b6d0c719f21bb054cc0"
server_key_iv: "35e20c04cb6f4f928001099d727d6f44"
server_key_version: "encryptionkey@0"
I20260618 22:39:15.392033 12188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "ebff97d8cce5416a95db1cf751e0018b"
format_stamp: "Formatted at 2026-06-18 22:39:15 on dist-test-slave-lnj3"
server_key: "973d91ce162a5b6d0c719f21bb054cc0"
server_key_iv: "35e20c04cb6f4f928001099d727d6f44"
server_key_version: "encryptionkey@0"
I20260618 22:39:15.396847 12188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260618 22:39:15.400091 12203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:15.401377 12188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260618 22:39:15.401501 12188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "ebff97d8cce5416a95db1cf751e0018b"
format_stamp: "Formatted at 2026-06-18 22:39:15 on dist-test-slave-lnj3"
server_key: "973d91ce162a5b6d0c719f21bb054cc0"
server_key_iv: "35e20c04cb6f4f928001099d727d6f44"
server_key_version: "encryptionkey@0"
I20260618 22:39:15.401648 12188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:15.412182 12188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:15.416051 12188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:15.416289 12188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:15.417112 12188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:15.418313 12188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:15.418380 12188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:15.418462 12188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:15.418502 12188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:15.446333 12188 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:43029
I20260618 22:39:15.446378 12315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:43029 every 8 connection(s)
I20260618 22:39:15.447532 12188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:39:15.456820 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 12188
I20260618 22:39:15.456988 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:39:15.457762 5975 external_mini_cluster.cc:1428] Setting key bd17bbe43c007147265bb50b912f66ea
Jun 18 22:39:15 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822355, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM
I20260618 22:39:15.466219 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:15.449830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:59057 (local address 127.5.213.254:33087)
0618 22:39:15.450243 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:15.450249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:15.451183 (+ 934us) server_negotiation.cc:407] Connection header received
0618 22:39:15.452902 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:15.452906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:15.452977 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:15.453074 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:15.454400 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:15.455129 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:15.456568 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:15.456969 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:15.457075 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:15.460631 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:15.460637 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:15.460643 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:15.460679 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:15.463449 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:15.464106 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:15.464111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:15.464115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:15.464237 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:15.464578 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:15.464582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:15.464585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:15.464881 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:15.465004 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:15.465644 (+ 640us) server_negotiation.cc:299] Negotiation successful
0618 22:39:15.465873 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":191,"threads_started":1}
I20260618 22:39:15.467160 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:15.450379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.193:59057)
0618 22:39:15.451039 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:15.451075 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:15.452684 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:15.453252 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:15.453259 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:15.453745 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:15.454270 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:15.454287 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:15.455247 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:15.455250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:15.456436 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:15.456450 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:15.457183 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:15.457191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:15.457462 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:15.458126 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:15.458153 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:15.460431 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:15.463586 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:15.463593 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:15.463605 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:15.463998 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:15.464329 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:15.464333 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:15.464335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:15.464496 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:15.464981 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:15.464987 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:15.465296 (+ 309us) client_negotiation.cc:769] Sending connection context
0618 22:39:15.465534 (+ 238us) client_negotiation.cc:240] Negotiation successful
0618 22:39:15.466028 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":269,"threads_started":1}
I20260618 22:39:15.468602 12316 heartbeater.cc:347] Connected to a master server at 127.5.213.254:33087
I20260618 22:39:15.468914 12316 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:15.469667 12316 heartbeater.cc:511] Master 127.5.213.254:33087 requested a full tablet report, sending...
I20260618 22:39:15.471923 12127 ts_manager.cc:194] Registered new tserver with Master: ebff97d8cce5416a95db1cf751e0018b (127.5.213.193:43029)
I20260618 22:39:15.473853 12127 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:59057
WARNING: no policy specified for oryx/127.5.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.194.keytab.
Entry for principal oryx/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.194.keytab.
I20260618 22:39:15.502337 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:33087
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.194.keytab
--principal=oryx/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:15.811069 12322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:15.811359 12322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:15.811475 12322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:15.818681 12322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:15.818790 12322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:15.818930 12322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:39:15.828684 12322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.194.keytab
--principal=oryx/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:33087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:15.829921 12322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:15.831278 12322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:15.843519 12327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:15.843524 12328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:15.843502 12330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:15.844194 12322 server_base.cc:1034] running on GCE node
I20260618 22:39:15.844748 12322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:15.845578 12322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:15.846747 12322 hybrid_clock.cc:648] HybridClock initialized: now 1781822355846690 us; error 43 us; skew 500 ppm
I20260618 22:39:15.847047 12322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:15 dist-test-slave-lnj3 krb5kdc[12082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822355, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:15.849993 12322 init.cc:377] Logged in from keytab as oryx/127.5.213.194@KRBTEST.COM (short username oryx)
I20260618 22:39:15.851320 12322 webserver.cc:460] Webserver started at http://127.5.213.194:40319/ using document root <none> and password file <none>
I20260618 22:39:15.851953 12322 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:15.852068 12322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:15.852352 12322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:15.855139 12322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "639c8bb1736b47da9fe60e4606c1f654"
format_stamp: "Formatted at 2026-06-18 22:39:15 on dist-test-slave-lnj3"
server_key: "034b008d35e5c9e453cce73814f5d74a"
server_key_iv: "63d31ab00135c94ff72bf52530f3531e"
server_key_version: "encryptionkey@0"
I20260618 22:39:15.855798 12322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "639c8bb1736b47da9fe60e4606c1f654"
format_stamp: "Formatted at 2026-06-18 22:39:15 on dist-test-slave-lnj3"
server_key: "034b008d35e5c9e453cce73814f5d74a"
server_key_iv: "63d31ab00135c94ff72bf52530f3531e"
server_key_version: "encryptionkey@0"
I20260618 22:39:15.860658 12322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:39:15.863760 12337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:15.865166 12322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:39:15.865319 12322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "639c8bb1736b47da9fe60e4606c1f654"
format_stamp: "Formatted at 2026-06-18 22:39:15 on dist-test-slave-lnj3"
server_key: "034b008d35e5c9e453cce73814f5d74a"
server_key_iv: "63d31ab00135c94ff72bf52530f3531e"
server_key_version: "encryptionkey@0"
I20260618 22:39:15.865482 12322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:15.899861 12322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:15.903831 12322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:15.904096 12322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:15.905037 12322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:15.906389 12322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:15.906463 12322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:15.906581 12322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:15.906651 12322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:15.935621 12322 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:43543
I20260618 22:39:15.935664 12449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:43543 every 8 connection(s)
I20260618 22:39:15.936867 12322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:39:15.939261 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 12322
I20260618 22:39:15.939451 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:39:15.939872 5975 external_mini_cluster.cc:1428] Setting key 29612aa71fcfe3ce79e6cd123edffd60
Jun 18 22:39:15 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822355, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM
I20260618 22:39:15.956292 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:15.939244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:35207 (local address 127.5.213.254:33087)
0618 22:39:15.939442 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:15.939449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:15.940657 (+ 1208us) server_negotiation.cc:407] Connection header received
0618 22:39:15.942556 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:15.942562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:15.942672 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:15.942783 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:15.944123 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:15.945079 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:15.946511 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:15.947129 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:15.947239 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:15.950913 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:15.950917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:15.950921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:15.950950 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:15.953390 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:15.954194 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:15.954197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:15.954200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:15.954333 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:15.954771 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:15.954774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:15.954776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:15.955003 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:15.955108 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:15.955891 (+ 783us) server_negotiation.cc:299] Negotiation successful
0618 22:39:15.956044 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:39:15.957763 12452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:15.939782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.194:35207)
0618 22:39:15.940518 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:15.940561 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:15.942304 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:15.942957 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:15.942964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:15.943430 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:15.943971 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:15.943988 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:15.945203 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:15.945207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:15.946385 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:15.946395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:15.947350 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:15.947357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:15.947566 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:15.948188 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:15.948219 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:15.950715 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:15.953511 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:15.953519 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:15.953535 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:15.954084 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:15.954423 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:15.954427 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:15.954431 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:15.954683 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:15.955122 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:15.955130 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:15.955541 (+ 411us) client_negotiation.cc:769] Sending connection context
0618 22:39:15.955850 (+ 309us) client_negotiation.cc:240] Negotiation successful
0618 22:39:15.956381 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":304,"threads_started":1}
WARNING: no policy specified for oryx/127.5.213.195@KRBTEST.COM; defaulting to no policy
I20260618 22:39:15.959229 12450 heartbeater.cc:347] Connected to a master server at 127.5.213.254:33087
I20260618 22:39:15.959529 12450 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:15.960187 12450 heartbeater.cc:511] Master 127.5.213.254:33087 requested a full tablet report, sending...
I20260618 22:39:15.961436 12127 ts_manager.cc:194] Registered new tserver with Master: 639c8bb1736b47da9fe60e4606c1f654 (127.5.213.194:43543)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.213.195@KRBTEST.COM" created.
I20260618 22:39:15.962292 12127 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:35207
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.195.keytab.
Entry for principal oryx/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.195.keytab.
I20260618 22:39:15.981567 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:33087
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.195.keytab
--principal=oryx/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:16.291716 12455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:16.291970 12455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:16.292085 12455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:16.299067 12455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:16.299165 12455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:16.299305 12455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:39:16.308969 12455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.195.keytab
--principal=oryx/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:33087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:16.310148 12455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:16.311408 12455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:16.323230 12461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:16.323371 12463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:16.323321 12460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:16.324105 12455 server_base.cc:1034] running on GCE node
I20260618 22:39:16.324612 12455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:16.325603 12455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:16.326829 12455 hybrid_clock.cc:648] HybridClock initialized: now 1781822356326781 us; error 35 us; skew 500 ppm
I20260618 22:39:16.327123 12455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:16 dist-test-slave-lnj3 krb5kdc[12082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822356, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:16.330009 12455 init.cc:377] Logged in from keytab as oryx/127.5.213.195@KRBTEST.COM (short username oryx)
I20260618 22:39:16.331329 12455 webserver.cc:460] Webserver started at http://127.5.213.195:33847/ using document root <none> and password file <none>
I20260618 22:39:16.331908 12455 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:16.331998 12455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:16.332237 12455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:16.334933 12455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "b834563cb2c64ddeb15499cbd0c68a4c"
format_stamp: "Formatted at 2026-06-18 22:39:16 on dist-test-slave-lnj3"
server_key: "de28b4aec6603f619847b791ff1a2135"
server_key_iv: "b249bba5007881b37bfe8eac042bef04"
server_key_version: "encryptionkey@0"
I20260618 22:39:16.335604 12455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "b834563cb2c64ddeb15499cbd0c68a4c"
format_stamp: "Formatted at 2026-06-18 22:39:16 on dist-test-slave-lnj3"
server_key: "de28b4aec6603f619847b791ff1a2135"
server_key_iv: "b249bba5007881b37bfe8eac042bef04"
server_key_version: "encryptionkey@0"
I20260618 22:39:16.340294 12455 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260618 22:39:16.343394 12470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:16.344604 12455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:39:16.344724 12455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "b834563cb2c64ddeb15499cbd0c68a4c"
format_stamp: "Formatted at 2026-06-18 22:39:16 on dist-test-slave-lnj3"
server_key: "de28b4aec6603f619847b791ff1a2135"
server_key_iv: "b249bba5007881b37bfe8eac042bef04"
server_key_version: "encryptionkey@0"
I20260618 22:39:16.344877 12455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:16.361608 12455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:16.365506 12455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:16.365727 12455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:16.366539 12455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:16.367796 12455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:16.367863 12455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:16.367974 12455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:16.368026 12455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:16.396552 12455 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:43255
I20260618 22:39:16.396574 12582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:43255 every 8 connection(s)
I20260618 22:39:16.397823 12455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:39:16.398756 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 12455
I20260618 22:39:16.398929 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:39:16.399382 5975 external_mini_cluster.cc:1428] Setting key f4029e84ec4a154bb26d9dbbd5300b1f
Jun 18 22:39:16 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822356, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM
I20260618 22:39:16.416831 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:16.400416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:60853 (local address 127.5.213.254:33087)
0618 22:39:16.400568 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:16.400573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:16.401936 (+ 1363us) server_negotiation.cc:407] Connection header received
0618 22:39:16.403828 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:16.403832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:16.403895 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:16.403970 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:16.405455 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.406178 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.407757 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.408147 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.408237 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:16.411430 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:16.411435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:16.411438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:16.411469 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:16.414314 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:16.415041 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:16.415044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:16.415047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:16.415155 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:16.415489 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:16.415492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:16.415494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:16.415741 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:16.415843 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:16.416449 (+ 606us) server_negotiation.cc:299] Negotiation successful
0618 22:39:16.416602 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:39:16.417699 12585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:16.400923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.195:60853)
0618 22:39:16.401767 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:16.401825 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:16.403600 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:16.404183 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:16.404191 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:16.404761 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:16.405318 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:16.405335 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.406313 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.406317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:16.407628 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:16.407640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.408346 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.408351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:16.408555 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:16.409120 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:16.409147 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:16.411251 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:16.414442 (+ 3191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:16.414452 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:16.414466 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:16.414940 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:16.415234 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:16.415237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:16.415239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:16.415407 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:16.415867 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:16.415873 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:16.416175 (+ 302us) client_negotiation.cc:769] Sending connection context
0618 22:39:16.416397 (+ 222us) client_negotiation.cc:240] Negotiation successful
0618 22:39:16.416734 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":349,"threads_started":1}
I20260618 22:39:16.418821 12583 heartbeater.cc:347] Connected to a master server at 127.5.213.254:33087
I20260618 22:39:16.419060 12583 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:16.419687 12583 heartbeater.cc:511] Master 127.5.213.254:33087 requested a full tablet report, sending...
I20260618 22:39:16.420892 12127 ts_manager.cc:194] Registered new tserver with Master: b834563cb2c64ddeb15499cbd0c68a4c (127.5.213.195:43255)
I20260618 22:39:16.421587 12127 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:60853
I20260618 22:39:16.426393 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.213.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.213.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.213.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.253.keytab.
Entry for principal oryx/127.5.213.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.253.keytab.
I20260618 22:39:16.467919 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.253:46277
--webserver_interface=127.5.213.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.253.keytab
--principal=oryx/127.5.213.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:33087,127.5.213.253:46277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:39:16.477301 12316 heartbeater.cc:503] Master 127.5.213.254:33087 was elected leader, sending a full tablet report...
W20260618 22:39:16.769754 12588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:16.770023 12588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:16.770138 12588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:16.777242 12588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:39:16.777339 12588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:16.777384 12588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:16.777438 12588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:39:16.777479 12588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:39:16.786875 12588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:33087,127.5.213.253:46277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.253.keytab
--principal=oryx/127.5.213.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.253:46277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.213.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
W20260618 22:39:16.787175 12588 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.5.213.254:33087,127.5.213.253:46277'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260618 22:39:16.788100 12588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:16.789315 12588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:16.800388 12593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:16.800468 12596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:16.800449 12594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:16.801046 12588 server_base.cc:1034] running on GCE node
I20260618 22:39:16.801797 12588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:16.803174 12588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:16.804409 12588 hybrid_clock.cc:648] HybridClock initialized: now 1781822356804334 us; error 53 us; skew 500 ppm
I20260618 22:39:16.804760 12588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:16 dist-test-slave-lnj3 krb5kdc[12082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822356, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:16.807724 12588 init.cc:377] Logged in from keytab as oryx/127.5.213.253@KRBTEST.COM (short username oryx)
I20260618 22:39:16.808980 12588 webserver.cc:460] Webserver started at http://127.5.213.253:37489/ using document root <none> and password file <none>
I20260618 22:39:16.809588 12588 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:16.809682 12588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:16.809953 12588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:16.812688 12588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/data/instance:
uuid: "b13db14e31f04ab08280cf6a17ff0810"
format_stamp: "Formatted at 2026-06-18 22:39:16 on dist-test-slave-lnj3"
server_key: "1c45d65dea9e0c64194a13791b1602e9"
server_key_iv: "5c9ce2e3fca849c4b269e0573810f91e"
server_key_version: "encryptionkey@0"
I20260618 22:39:16.813362 12588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/wal/instance:
uuid: "b13db14e31f04ab08280cf6a17ff0810"
format_stamp: "Formatted at 2026-06-18 22:39:16 on dist-test-slave-lnj3"
server_key: "1c45d65dea9e0c64194a13791b1602e9"
server_key_iv: "5c9ce2e3fca849c4b269e0573810f91e"
server_key_version: "encryptionkey@0"
I20260618 22:39:16.817986 12588 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260618 22:39:16.821082 12603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:16.822288 12588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:39:16.822413 12588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/wal
uuid: "b13db14e31f04ab08280cf6a17ff0810"
format_stamp: "Formatted at 2026-06-18 22:39:16 on dist-test-slave-lnj3"
server_key: "1c45d65dea9e0c64194a13791b1602e9"
server_key_iv: "5c9ce2e3fca849c4b269e0573810f91e"
server_key_version: "encryptionkey@0"
I20260618 22:39:16.822584 12588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:16.840811 12588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:16.845992 12588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:16.846253 12588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:16.868363 12654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.253:46277 every 8 connection(s)
I20260618 22:39:16.868407 12588 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.253:46277
I20260618 22:39:16.870116 12588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/data/info.pb
I20260618 22:39:16.873567 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 12588
I20260618 22:39:16.873893 12655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:16.883558 12655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:39:16 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822356, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.253@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM
Jun 18 22:39:16 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.213.253@KRBTEST.COM
I20260618 22:39:16.905973 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:16.886905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48328 (local address 127.5.213.254:33087)
0618 22:39:16.887064 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:16.887070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:16.888817 (+ 1747us) server_negotiation.cc:407] Connection header received
0618 22:39:16.890496 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:16.890502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:16.890578 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:16.890683 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:16.892305 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.893411 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.895622 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.896217 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.896346 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:16.899268 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:16.899276 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:16.899281 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:16.899323 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:16.902783 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:16.903665 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:16.903669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:16.903671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:16.903814 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:16.904228 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:16.904233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:16.904235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:16.904629 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:16.904751 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:16.905404 (+ 653us) server_negotiation.cc:299] Negotiation successful
0618 22:39:16.905603 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 22:39:16.907043 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:16.887160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.0.0.1:48328)
0618 22:39:16.888664 (+ 1504us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:16.888715 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:16.890175 (+ 1460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:16.890774 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:16.890782 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:16.891605 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:16.892111 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:16.892136 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.893552 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.893559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:16.895481 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:16.895496 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.896420 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.896426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:16.896560 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:16.897075 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:16.897095 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:16.899061 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:16.902931 (+ 3870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:16.902946 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:16.902959 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:16.903535 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:16.903907 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:16.903912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:16.903916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:16.904128 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:16.904747 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:16.904755 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:16.905055 (+ 300us) client_negotiation.cc:769] Sending connection context
0618 22:39:16.905276 (+ 221us) client_negotiation.cc:240] Negotiation successful
0618 22:39:16.905741 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1253,"mutex_wait_us":3,"thread_start_us":1121,"threads_started":1}
I20260618 22:39:16.907591 12658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:16.875746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43604 (local address 127.5.213.253:46277)
0618 22:39:16.887571 (+ 11825us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:16.887586 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:16.888357 (+ 771us) server_negotiation.cc:407] Connection header received
0618 22:39:16.890150 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:16.890171 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:16.890405 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:16.890571 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:16.892137 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.893718 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.894965 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.895661 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.895839 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:16.900507 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:16.900515 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:16.900525 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:16.900555 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:16.903065 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:16.903936 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:16.903942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:16.903944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:16.904081 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:16.904472 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:16.904477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:16.904480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:16.905170 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:16.905481 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:16.906757 (+ 1276us) server_negotiation.cc:299] Negotiation successful
0618 22:39:16.907161 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":11586,"spinlock_wait_cycles":29184,"thread_start_us":11749,"threads_started":1}
I20260618 22:39:16.908773 12655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:39:16 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822356, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.253@KRBTEST.COM for oryx/127.5.213.253@KRBTEST.COM
I20260618 22:39:16.922147 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:16.909434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:46277 (local address 127.0.0.1:43614)
0618 22:39:16.909979 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:16.909995 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:16.910089 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:16.910499 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:16.910501 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:16.910832 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:16.910942 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:16.910949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.912277 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.912281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:16.913101 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:16.913111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.913945 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.913949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:16.914122 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:16.914566 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:16.914581 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:16.916613 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:16.920097 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:16.920101 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:16.920103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:16.920443 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:16.920788 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:16.920791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:16.920793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:16.920930 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:16.921527 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:16.921530 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:16.921630 (+ 100us) client_negotiation.cc:769] Sending connection context
0618 22:39:16.921734 (+ 104us) client_negotiation.cc:240] Negotiation successful
0618 22:39:16.921922 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":3}
I20260618 22:39:16.922601 12658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:16.909580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43614 (local address 127.5.213.253:46277)
0618 22:39:16.909717 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:16.909722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:16.910264 (+ 542us) server_negotiation.cc:407] Connection header received
0618 22:39:16.910299 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:16.910301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:16.910346 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:16.910396 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:16.911064 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.912138 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.913227 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.913818 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.913944 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:16.916792 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:16.916798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:16.916803 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:16.916839 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:16.919951 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:16.920548 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:16.920553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:16.920557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:16.920692 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:16.921029 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:16.921034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:16.921038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:16.921406 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:16.921550 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:16.921847 (+ 297us) server_negotiation.cc:299] Negotiation successful
0618 22:39:16.922072 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":52}
I20260618 22:39:16.927033 12655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: Bootstrap starting.
I20260618 22:39:16.930040 12655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:16.931056 12655 log.cc:824] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:16.933560 12655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: No bootstrap required, opened a new log
I20260618 22:39:16.942482 12655 raft_consensus.cc:348] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } }
I20260618 22:39:16.942809 12655 raft_consensus.cc:374] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:16.942899 12655 raft_consensus.cc:729] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b13db14e31f04ab08280cf6a17ff0810, State: Initialized, Role: FOLLOWER
I20260618 22:39:16.943440 12655 consensus_queue.cc:260] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } }
I20260618 22:39:16.944345 12661 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } } }
I20260618 22:39:16.944710 12661 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:39:16.945137 12655 sys_catalog.cc:564] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: configured and running, proceeding with master startup.
W20260618 22:39:16.953809 12672 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:39:16.953969 12672 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:39:16.955892 12588 master_runner.cc:428] Detected that this master b13db14e31f04ab08280cf6a17ff0810 is joining an existing cluster
I20260618 22:39:16.955971 12588 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.213.253:46277
I20260618 22:39:16.965083 12450 heartbeater.cc:503] Master 127.5.213.254:33087 was elected leader, sending a full tablet report...
I20260618 22:39:16.976322 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:16.959966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48344 (local address 127.5.213.254:33087)
0618 22:39:16.960134 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:16.960140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:16.961214 (+ 1074us) server_negotiation.cc:407] Connection header received
0618 22:39:16.961473 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:16.961479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:16.961554 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:16.961620 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:16.963024 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.964171 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.965950 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.966749 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.966844 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:16.969417 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:16.969424 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:16.969430 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:16.969477 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:16.973171 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:16.973919 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:16.973924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:16.973928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:16.974080 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:16.974533 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:16.974537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:16.974541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:16.974923 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:16.975058 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:16.975691 (+ 633us) server_negotiation.cc:299] Negotiation successful
0618 22:39:16.975922 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 22:39:16.977385 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:16.960036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.0.0.1:48344)
0618 22:39:16.961119 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:16.961137 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:16.961327 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:16.961815 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:16.961820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:16.962451 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:16.962856 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:16.962868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.964352 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.964357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:16.965790 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:16.965806 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.967142 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.967149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:16.967326 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:16.967960 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:16.967982 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:16.969149 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:16.973345 (+ 4196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:16.973349 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:16.973354 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:16.973788 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:16.974220 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:16.974224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:16.974228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:16.974400 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:16.975150 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:16.975155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:16.975377 (+ 222us) client_negotiation.cc:769] Sending connection context
0618 22:39:16.976562 (+ 1185us) client_negotiation.cc:240] Negotiation successful
0618 22:39:16.976927 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":369,"threads_started":1}
WARNING: no policy specified for oryx/127.5.213.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.213.252@KRBTEST.COM" created.
I20260618 22:39:16.991269 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:16.960935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:46277 (local address 127.0.0.1:43616)
0618 22:39:16.977318 (+ 16383us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:16.977339 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:16.977485 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:16.977854 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:16.977861 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:16.978453 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:16.978746 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:16.978758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.980115 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.980123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:16.981632 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:16.981647 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.982903 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.982909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:16.983069 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:16.983619 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:16.983640 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:16.984621 (+ 981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:16.988494 (+ 3873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:16.988498 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:16.988502 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:16.988992 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:16.989404 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:16.989408 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:16.989410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:16.989542 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:16.990257 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:16.990262 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:16.990384 (+ 122us) client_negotiation.cc:769] Sending connection context
0618 22:39:16.990499 (+ 115us) client_negotiation.cc:240] Negotiation successful
0618 22:39:16.990785 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16265,"thread_start_us":16087,"threads_started":1}
I20260618 22:39:16.992092 12658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:16.960865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43616 (local address 127.5.213.253:46277)
0618 22:39:16.961023 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:16.961029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:16.977409 (+ 16380us) server_negotiation.cc:407] Connection header received
0618 22:39:16.977608 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:16.977612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:16.977668 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:16.977719 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:16.978886 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.979926 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.981777 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:16.982496 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:16.982599 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:16.984814 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:16.984820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:16.984825 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:16.984856 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:16.988386 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:16.989124 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:16.989129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:16.989132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:16.989277 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:16.989724 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:16.989728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:16.989731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:16.990089 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:16.990216 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:16.991623 (+ 1407us) server_negotiation.cc:299] Negotiation successful
0618 22:39:16.991802 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.213.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.252.keytab.
Entry for principal oryx/127.5.213.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.252.keytab.
I20260618 22:39:17.021836 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.252:37039
--webserver_interface=127.5.213.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.252.keytab
--principal=oryx/127.5.213.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:33087,127.5.213.253:46277,127.5.213.252:37039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:39:17 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822354, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.254@KRBTEST.COM for oryx/127.5.213.253@KRBTEST.COM
I20260618 22:39:17.032863 12658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:16.984282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43618 (local address 127.5.213.253:46277)
0618 22:39:16.992286 (+ 8004us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:16.992291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:17.011762 (+ 19471us) server_negotiation.cc:407] Connection header received
0618 22:39:17.012160 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:17.012166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:17.012285 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:17.012476 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:17.014115 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.015440 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.017731 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.018523 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.018700 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.025341 (+ 6641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:17.025348 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:17.025353 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:17.025392 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:17.029108 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.030086 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.030092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.030095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.030245 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.031091 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.031097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.031101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.031466 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:17.031592 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:17.032253 (+ 661us) server_negotiation.cc:299] Negotiation successful
0618 22:39:17.032488 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7914,"thread_start_us":26655,"threads_started":1}
I20260618 22:39:17.034197 12681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:16.983945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:46277 (local address 127.0.0.1:43618)
0618 22:39:17.011563 (+ 27618us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:17.011620 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:17.011859 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:17.012733 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:17.012742 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:17.013642 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:17.013932 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.013945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.015617 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.015624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.017580 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.017596 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.018829 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.018837 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.019043 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.019860 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:17.019890 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:17.025059 (+ 5169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:17.029346 (+ 4287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:17.029357 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:17.029365 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:17.029943 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:17.030401 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:17.030406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:17.030410 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:17.030954 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:17.031702 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:17.031710 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:17.031918 (+ 208us) client_negotiation.cc:769] Sending connection context
0618 22:39:17.033359 (+ 1441us) client_negotiation.cc:240] Negotiation successful
0618 22:39:17.033783 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27331,"thread_start_us":28129,"threads_started":1}
I20260618 22:39:17.037114 12127 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.213.253:46277: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810"
member_type: NON_VOTER
last_known_addr {
host: "127.5.213.253"
port: 46277
}
attrs {
promote: true
}
}
dest_uuid: "798ea03554f649e8b59eff0c77e07c60"
cas_config_opid_index: -1
I20260618 22:39:17.038303 12127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: NON_VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: true } }
I20260618 22:39:17.055277 12630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:17.054701 12687 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b13db14e31f04ab08280cf6a17ff0810 (127.5.213.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: NON_VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: true } } }
I20260618 22:39:17.059103 12630 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 1 FOLLOWER]: Refusing update from remote peer 798ea03554f649e8b59eff0c77e07c60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260618 22:39:17.060477 12686 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: NON_VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:17.074851 12687 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.213.253:46277
I20260618 22:39:17.075546 12588 master.cc:537] Master@127.5.213.253:46277 shutting down...
I20260618 22:39:17.084919 12688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: NON_VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: true } } }
I20260618 22:39:17.085263 12688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:17.125960 12630 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b13db14e31f04ab08280cf6a17ff0810 (127.5.213.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: NON_VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: true } } }
I20260618 22:39:17.137916 12661 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 798ea03554f649e8b59eff0c77e07c60. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: NON_VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: true } } }
I20260618 22:39:17.138206 12661 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: This master's current role is: LEARNER
I20260618 22:39:17.137923 12692 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: NON_VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: true } } }
I20260618 22:39:17.138362 12692 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: This master's current role is: LEARNER
I20260618 22:39:17.139701 12695 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: NON_VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: true } } }
I20260618 22:39:17.139904 12695 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: This master's current role is: LEARNER
I20260618 22:39:17.168720 12588 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 1 LEARNER]: Raft consensus shutting down.
I20260618 22:39:17.169284 12588 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 1 LEARNER]: Raft consensus is shut down!
I20260618 22:39:17.169454 12588 tablet_replica.cc:330] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: stopping tablet replica
I20260618 22:39:17.187500 12588 master.cc:552] Master@127.5.213.253:46277 shutdown complete.
I20260618 22:39:17.187598 12588 master_runner.cc:305] Clearing existing system tablet
I20260618 22:39:17.187997 12588 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:17.188100 12588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:17.191447 12588 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.003s
I20260618 22:39:17.193125 12697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:17.193514 12588 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:17.193953 12588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260618 22:39:17.194092 12588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/wal
uuid: "b13db14e31f04ab08280cf6a17ff0810"
format_stamp: "Formatted at 2026-06-18 22:39:16 on dist-test-slave-lnj3"
server_key: "1c45d65dea9e0c64194a13791b1602e9"
server_key_iv: "5c9ce2e3fca849c4b269e0573810f91e"
server_key_version: "encryptionkey@0"
I20260618 22:39:17.198825 12588 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:17.205850 12588 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260618 22:39:17.206030 12588 log.cc:1196] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260618 22:39:17.206501 12588 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: Deleting consensus metadata
I20260618 22:39:17.206785 12588 master_runner.cc:315] Copying system tablet from 127.5.213.254:33087
I20260618 22:39:17.219108 12588 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: tablet copy: Beginning tablet copy session from remote peer at address 127.5.213.254:33087
I20260618 22:39:17.234336 12703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.219991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.0.0.1:48358)
0618 22:39:17.220554 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:17.220578 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:17.220732 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:17.221158 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:17.221163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:17.221693 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:17.221960 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.221972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.223498 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.223503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.225027 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.225044 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.225970 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.225976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.226178 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.226806 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:17.226826 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:17.227834 (+ 1008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:17.231656 (+ 3822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:17.231661 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:17.231665 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:17.232134 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:17.232510 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:17.232515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:17.232519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:17.232668 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:17.233243 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:17.233248 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:17.233419 (+ 171us) client_negotiation.cc:769] Sending connection context
0618 22:39:17.233578 (+ 159us) client_negotiation.cc:240] Negotiation successful
0618 22:39:17.233871 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":293,"threads_started":1}
I20260618 22:39:17.234720 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.219976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.5.213.254:33087)
0618 22:39:17.220178 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:17.220184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:17.220781 (+ 597us) server_negotiation.cc:407] Connection header received
0618 22:39:17.220906 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:17.220912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:17.221004 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:17.221139 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:17.222163 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.223356 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.225200 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.225806 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.225930 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.228045 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:17.228052 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:17.228057 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:17.228106 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:17.231536 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.232254 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.232260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.232264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.232406 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.232775 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.232779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.232783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.233128 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:17.233278 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:17.233741 (+ 463us) server_negotiation.cc:299] Negotiation successful
0618 22:39:17.234101 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260618 22:39:17.235569 12147 tablet_copy_service.cc:140] P 798ea03554f649e8b59eff0c77e07c60: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b13db14e31f04ab08280cf6a17ff0810 ({username='oryx', principal='oryx/127.5.213.253@KRBTEST.COM'} at 127.0.0.1:48358)
I20260618 22:39:17.235841 12147 tablet_copy_service.cc:161] P 798ea03554f649e8b59eff0c77e07c60: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b13db14e31f04ab08280cf6a17ff0810 at {username='oryx', principal='oryx/127.5.213.253@KRBTEST.COM'} at 127.0.0.1:48358: session id = b13db14e31f04ab08280cf6a17ff0810-00000000000000000000000000000000
I20260618 22:39:17.240162 12147 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60: Tablet Copy: opened 0 blocks and 1 log segments
I20260618 22:39:17.242377 12588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260618 22:39:17.248831 12588 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: tablet copy: Starting download of 0 data blocks...
I20260618 22:39:17.249096 12588 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: tablet copy: Starting download of 1 WAL segments...
I20260618 22:39:17.251593 12588 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260618 22:39:17.255743 12147 tablet_copy_service.cc:342] P 798ea03554f649e8b59eff0c77e07c60: Request end of tablet copy session b13db14e31f04ab08280cf6a17ff0810-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.213.253@KRBTEST.COM'} at 127.0.0.1:48358
I20260618 22:39:17.255964 12147 tablet_copy_service.cc:433] P 798ea03554f649e8b59eff0c77e07c60: ending tablet copy session b13db14e31f04ab08280cf6a17ff0810-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b13db14e31f04ab08280cf6a17ff0810
I20260618 22:39:17.266417 12588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:17.268994 12704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:17.273756 12705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:17.280217 12707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:17.281518 12588 server_base.cc:1034] running on GCE node
I20260618 22:39:17.281872 12588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:17.282541 12588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:17.283751 12588 hybrid_clock.cc:648] HybridClock initialized: now 1781822357283685 us; error 40 us; skew 500 ppm
I20260618 22:39:17.283990 12588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:17 dist-test-slave-lnj3 krb5kdc[12082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822357, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:17.286093 12588 init.cc:377] Logged in from keytab as oryx/127.5.213.253@KRBTEST.COM (short username oryx)
I20260618 22:39:17.287441 12588 webserver.cc:460] Webserver started at http://127.5.213.253:45869/ using document root <none> and password file <none>
I20260618 22:39:17.287806 12588 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:17.287889 12588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:17.291056 12588 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:39:17.292984 12714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:17.293390 12588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260618 22:39:17.293560 12588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/wal
uuid: "b13db14e31f04ab08280cf6a17ff0810"
format_stamp: "Formatted at 2026-06-18 22:39:16 on dist-test-slave-lnj3"
server_key: "1c45d65dea9e0c64194a13791b1602e9"
server_key_iv: "5c9ce2e3fca849c4b269e0573810f91e"
server_key_version: "encryptionkey@0"
I20260618 22:39:17.293808 12588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:17.315676 12588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:17.320001 12588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:17.339440 12588 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.253:46277
I20260618 22:39:17.339469 12765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.253:46277 every 8 connection(s)
I20260618 22:39:17.340739 12588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-1/data/info.pb
I20260618 22:39:17.346110 12766 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:39:17.350003 12766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: Bootstrap starting.
I20260618 22:39:17.363811 12766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:39:17.364351 12766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: Bootstrap complete.
I20260618 22:39:17.365407 12766 raft_consensus.cc:348] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: NON_VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: true } }
I20260618 22:39:17.365669 12766 raft_consensus.cc:729] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b13db14e31f04ab08280cf6a17ff0810, State: Initialized, Role: LEARNER
I20260618 22:39:17.365931 12766 consensus_queue.cc:260] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: NON_VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: true } }
I20260618 22:39:17.367451 12769 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: NON_VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: true } } }
I20260618 22:39:17.367775 12766 sys_catalog.cc:564] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:39:17.367889 12769 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: This master's current role is: LEARNER
I20260618 22:39:17.380016 12780 catalog_manager.cc:1259] Loaded cluster ID: 5d782181ba1a42da86921e7b22f586f2
I20260618 22:39:17.380125 12780 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: loading cluster ID for follower catalog manager: success
I20260618 22:39:17.383790 12780 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: acquiring CA information for follower catalog manager: success
I20260618 22:39:17.386392 12780 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260618 22:39:17.418236 12684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:17.418488 12684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:17.418596 12684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260618 22:39:17.423851 12583 heartbeater.cc:503] Master 127.5.213.254:33087 was elected leader, sending a full tablet report...
W20260618 22:39:17.425995 12684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:39:17.426092 12684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:17.426172 12684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:17.426246 12684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:39:17.426304 12684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:39:17.436066 12684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:33087,127.5.213.253:46277,127.5.213.252:37039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.252.keytab
--principal=oryx/127.5.213.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.252:37039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.213.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:17.437500 12684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:17.438740 12684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:17.449406 12785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:17.449635 12786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:17.449761 12788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:17.450539 12684 server_base.cc:1034] running on GCE node
I20260618 22:39:17.451292 12684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:17.452693 12684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:17.453910 12684 hybrid_clock.cc:648] HybridClock initialized: now 1781822357453864 us; error 36 us; skew 500 ppm
I20260618 22:39:17.454222 12684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:17 dist-test-slave-lnj3 krb5kdc[12082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822357, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:17.457118 12684 init.cc:377] Logged in from keytab as oryx/127.5.213.252@KRBTEST.COM (short username oryx)
I20260618 22:39:17.458433 12684 webserver.cc:460] Webserver started at http://127.5.213.252:42389/ using document root <none> and password file <none>
I20260618 22:39:17.459064 12684 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:17.459163 12684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:17.459422 12684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:17.462077 12684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/data/instance:
uuid: "0d65f54c7db549a8baf9dd615270b424"
format_stamp: "Formatted at 2026-06-18 22:39:17 on dist-test-slave-lnj3"
server_key: "ebde671697b08444011b4c1830ccbfb0"
server_key_iv: "30b1dcfd412d9ebfc0bfa97c68b2582e"
server_key_version: "encryptionkey@0"
I20260618 22:39:17.462792 12684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/wal/instance:
uuid: "0d65f54c7db549a8baf9dd615270b424"
format_stamp: "Formatted at 2026-06-18 22:39:17 on dist-test-slave-lnj3"
server_key: "ebde671697b08444011b4c1830ccbfb0"
server_key_iv: "30b1dcfd412d9ebfc0bfa97c68b2582e"
server_key_version: "encryptionkey@0"
I20260618 22:39:17.467427 12684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260618 22:39:17.470666 12795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:17.471861 12684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:39:17.471987 12684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/wal
uuid: "0d65f54c7db549a8baf9dd615270b424"
format_stamp: "Formatted at 2026-06-18 22:39:17 on dist-test-slave-lnj3"
server_key: "ebde671697b08444011b4c1830ccbfb0"
server_key_iv: "30b1dcfd412d9ebfc0bfa97c68b2582e"
server_key_version: "encryptionkey@0"
I20260618 22:39:17.472136 12684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:17.486666 12684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:17.490479 12684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:17.490717 12684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:17.506558 12684 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.252:37039
I20260618 22:39:17.506597 12846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.252:37039 every 8 connection(s)
I20260618 22:39:17.507861 12684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/data/info.pb
I20260618 22:39:17.511303 12847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:17.516196 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 12684
I20260618 22:39:17.517683 12847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:39:17 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822357, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.252@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM
Jun 18 22:39:17 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.213.252@KRBTEST.COM
I20260618 22:39:17.535554 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.520479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.5.213.254:33087)
0618 22:39:17.520683 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:17.520689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:17.521269 (+ 580us) server_negotiation.cc:407] Connection header received
0618 22:39:17.521658 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:17.521663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:17.521737 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:17.521839 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:17.523067 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.523816 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.525352 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.525809 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.525897 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.529261 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:17.529268 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:17.529272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:17.529312 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:17.532369 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.533112 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.533117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.533121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.533253 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.533728 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.533732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.533734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.534034 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:17.534184 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:17.534979 (+ 795us) server_negotiation.cc:299] Negotiation successful
0618 22:39:17.535204 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260618 22:39:17.536850 12850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.518221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59462 (local address 127.5.213.252:37039)
0618 22:39:17.518951 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:17.518966 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:17.519012 (+ 46us) server_negotiation.cc:407] Connection header received
0618 22:39:17.521153 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:17.521179 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:17.521557 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:17.521730 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:17.523068 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.524281 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.525434 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.525940 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.526109 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.530261 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:17.530271 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:17.530288 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:17.530322 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:17.533400 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.534078 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.534087 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.534090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.534262 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.534660 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.534665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.534669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.534994 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:17.535270 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:17.535587 (+ 317us) server_negotiation.cc:299] Negotiation successful
0618 22:39:17.535812 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":512,"spinlock_wait_cycles":25600,"thread_start_us":316,"threads_started":1}
I20260618 22:39:17.536892 12851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.520413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.0.0.1:48368)
0618 22:39:17.521122 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:17.521164 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:17.521448 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:17.521906 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:17.521913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:17.522533 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:17.522911 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.522928 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.523950 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.523956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.525215 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.525226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.526009 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.526016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.526162 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.526950 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:17.526971 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:17.529071 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:17.532487 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:17.532496 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:17.532505 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:17.532994 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:17.533353 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:17.533356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:17.533358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:17.533555 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:17.534160 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:17.534166 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:17.534551 (+ 385us) client_negotiation.cc:769] Sending connection context
0618 22:39:17.534881 (+ 330us) client_negotiation.cc:240] Negotiation successful
0618 22:39:17.535693 (+ 812us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":4,"spinlock_wait_cycles":3328,"thread_start_us":324,"threads_started":1}
I20260618 22:39:17.538193 12847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:39:17 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822357, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.252@KRBTEST.COM for oryx/127.5.213.253@KRBTEST.COM
I20260618 22:39:17.550235 12851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.538780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:46277 (local address 127.0.0.1:43622)
0618 22:39:17.538940 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:17.538958 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:17.539043 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:17.539742 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:17.539745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:17.540027 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:17.540146 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.540153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.541433 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.541436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.542350 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.542367 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.543006 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.543010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.543137 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.543640 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:17.543656 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:17.545472 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:17.548034 (+ 2562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:17.548039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:17.548043 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:17.548445 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:17.548754 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:17.548759 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:17.548763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:17.548882 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:17.549351 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:17.549355 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:17.549493 (+ 138us) client_negotiation.cc:769] Sending connection context
0618 22:39:17.549601 (+ 108us) client_negotiation.cc:240] Negotiation successful
0618 22:39:17.549885 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260618 22:39:17.550268 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.538944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43622 (local address 127.5.213.253:46277)
0618 22:39:17.539480 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:17.539484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:17.539498 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:39:17.539549 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:17.539553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:17.539621 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:17.539708 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:17.540255 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.541299 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.542462 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.542908 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.542996 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.545674 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:17.545679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:17.545684 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:17.545715 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:17.547921 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.548544 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.548549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.548551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.548660 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.548973 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.548977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.548980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.549237 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:17.549387 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:17.549700 (+ 313us) server_negotiation.cc:299] Negotiation successful
0618 22:39:17.549890 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":295,"threads_started":1}
I20260618 22:39:17.551266 12847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:39:17 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822357, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.252@KRBTEST.COM for oryx/127.5.213.252@KRBTEST.COM
I20260618 22:39:17.561980 12851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.551936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:37039 (local address 127.0.0.1:59478)
0618 22:39:17.552084 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:17.552101 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:17.552186 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:17.552461 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:17.552466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:17.552752 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:17.552892 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.552899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.553802 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.553806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.554586 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.554595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.555196 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.555200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.555348 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.555795 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:17.555809 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:17.557553 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:17.560130 (+ 2577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:17.560134 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:17.560136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:17.560442 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:17.560732 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:17.560735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:17.560737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:17.560873 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:17.561267 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:17.561271 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:17.561373 (+ 102us) client_negotiation.cc:769] Sending connection context
0618 22:39:17.561479 (+ 106us) client_negotiation.cc:240] Negotiation successful
0618 22:39:17.561643 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260618 22:39:17.562021 12850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.552136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59478 (local address 127.5.213.252:37039)
0618 22:39:17.552257 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:17.552261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:17.552275 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:39:17.552308 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:17.552311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:17.552358 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:17.552425 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:17.552992 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.553699 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.554732 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.555084 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.555172 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.557749 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:17.557754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:17.557757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:17.557781 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:17.560036 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.560527 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.560530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.560533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.560630 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.560949 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.560952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.560954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.561182 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:17.561281 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:17.561554 (+ 273us) server_negotiation.cc:299] Negotiation successful
0618 22:39:17.561771 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":39}
I20260618 22:39:17.566470 12847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424: Bootstrap starting.
I20260618 22:39:17.569584 12847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:17.570556 12847 log.cc:824] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:17.573071 12847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424: No bootstrap required, opened a new log
I20260618 22:39:17.579893 12847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } }
I20260618 22:39:17.580173 12847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:17.580255 12847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d65f54c7db549a8baf9dd615270b424, State: Initialized, Role: FOLLOWER
I20260618 22:39:17.580773 12847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } }
I20260618 22:39:17.581487 12854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } } }
I20260618 22:39:17.581800 12854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:39:17.582412 12847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [sys.catalog]: configured and running, proceeding with master startup.
W20260618 22:39:17.590297 12865 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:39:17.590435 12865 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:39:17.591331 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 12188
I20260618 22:39:17.594758 12684 master_runner.cc:428] Detected that this master 0d65f54c7db549a8baf9dd615270b424 is joining an existing cluster
I20260618 22:39:17.594864 12684 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.213.252:37039
Jun 18 22:39:17 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.213.193@KRBTEST.COM
I20260618 22:39:17.613776 12867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.592336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47512 (local address 127.5.213.193:43029)
0618 22:39:17.592905 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:17.592914 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:17.592946 (+ 32us) server_negotiation.cc:407] Connection header received
0618 22:39:17.593016 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:17.593025 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:17.593260 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:17.593474 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:17.595197 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.596877 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.598903 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.599448 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.599562 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.605678 (+ 6116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:17.605695 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:17.605715 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:17.605760 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:17.609604 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.610456 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.610466 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.610470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.610646 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.611101 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.611107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.611110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.611536 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:17.611836 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:17.613070 (+ 1234us) server_negotiation.cc:299] Negotiation successful
0618 22:39:17.613356 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":248,"threads_started":1}
I20260618 22:39:17.615247 12311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:17.632606 12687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: NON_VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: true } } }
I20260618 22:39:17.632869 12687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:17.636021 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.600942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48384 (local address 127.5.213.254:33087)
0618 22:39:17.601126 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:17.601132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:17.621238 (+ 20106us) server_negotiation.cc:407] Connection header received
0618 22:39:17.621406 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:17.621412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:17.621504 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:17.621613 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:17.623120 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.624216 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.626185 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.626902 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.627000 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.629569 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:17.629576 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:17.629581 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:17.629634 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:17.632951 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.633704 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.633710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.633714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.633861 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.634292 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.634298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.634301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.634680 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:17.634788 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:17.635394 (+ 606us) server_negotiation.cc:299] Negotiation successful
0618 22:39:17.635649 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:39:17.636655 12877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.627924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:46277 (local address 127.0.0.1:43630)
0618 22:39:17.628308 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:17.628330 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:17.628433 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:17.629225 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:17.629230 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:17.629258 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:17.629520 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.629530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.631378 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.631383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.633569 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.633597 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.635486 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.635492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.635700 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.635770 (+ 70us) client_negotiation.cc:769] Sending connection context
0618 22:39:17.635835 (+ 65us) client_negotiation.cc:240] Negotiation successful
0618 22:39:17.635967 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":606,"threads_started":1}
I20260618 22:39:17.637005 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.601029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.0.0.1:48384)
0618 22:39:17.621106 (+ 20077us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:17.621129 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:17.621281 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:17.621864 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:17.621870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:17.622548 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:17.622944 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.622955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.624399 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.624404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.626026 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.626041 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.627320 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.627331 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.627524 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.628276 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:17.628300 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:17.629345 (+ 1045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:17.633096 (+ 3751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:17.633100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:17.633104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:17.633562 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:17.633997 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:17.634002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:17.634006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:17.634171 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:17.634873 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:17.634878 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:17.635057 (+ 179us) client_negotiation.cc:769] Sending connection context
0618 22:39:17.636247 (+ 1190us) client_negotiation.cc:240] Negotiation successful
0618 22:39:17.636566 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19915,"thread_start_us":26765,"threads_started":1}
I20260618 22:39:17.640086 12878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.628368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43630 (local address 127.5.213.253:46277)
0618 22:39:17.628871 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:17.628877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:17.628894 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:39:17.628956 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:17.628962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:17.629044 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:17.629813 (+ 769us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:17.630009 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.631213 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.633774 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.635302 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.637334 (+ 2032us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.639594 (+ 2260us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:17.639653 (+ 59us) server_negotiation.cc:299] Negotiation successful
0618 22:39:17.639790 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":391,"thread_start_us":293,"threads_started":1}
I20260618 22:39:17.641867 12769 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 798ea03554f649e8b59eff0c77e07c60. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: NON_VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: true } } }
I20260618 22:39:17.642050 12769 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: This master's current role is: LEARNER
I20260618 22:39:17.643530 12880 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60: attempting to promote NON_VOTER b13db14e31f04ab08280cf6a17ff0810 to VOTER
I20260618 22:39:17.644449 12880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } }
I20260618 22:39:17.646117 12741 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 1 LEARNER]: Refusing update from remote peer 798ea03554f649e8b59eff0c77e07c60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260618 22:39:17.646732 12880 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260618 22:39:17.647854 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.607298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:37039 (local address 127.0.0.1:59494)
0618 22:39:17.628030 (+ 20732us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:17.628050 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:17.628630 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:17.629343 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:17.629348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:17.629877 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:17.630166 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.630178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.633036 (+ 2858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.633041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.634407 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.634418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.638744 (+ 4326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.638749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.638991 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.639620 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:17.639644 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:17.640753 (+ 1109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:17.644812 (+ 4059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:17.644817 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:17.644820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:17.645228 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:17.645640 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:17.645644 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:17.645647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:17.645802 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:17.646783 (+ 981us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:17.646787 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:17.646993 (+ 206us) client_negotiation.cc:769] Sending connection context
0618 22:39:17.647148 (+ 155us) client_negotiation.cc:240] Negotiation successful
0618 22:39:17.647426 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20614,"mutex_wait_us":73,"spinlock_wait_cycles":1920,"thread_start_us":19898,"threads_started":1}
I20260618 22:39:17.648694 12850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.607103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59494 (local address 127.5.213.252:37039)
0618 22:39:17.607311 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:17.607317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:17.628121 (+ 20804us) server_negotiation.cc:407] Connection header received
0618 22:39:17.629015 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:17.629021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:17.629082 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:17.629150 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:17.631837 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.632836 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.637769 (+ 4933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.638409 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.638484 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.640914 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:17.640920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:17.640924 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:17.640953 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:17.644672 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.645342 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.645347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.645350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.645501 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.645928 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.645932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.645935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.646301 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:17.646735 (+ 434us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:17.648140 (+ 1405us) server_negotiation.cc:299] Negotiation successful
0618 22:39:17.648332 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260618 22:39:17.649449 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.601042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43626 (local address 127.5.213.253:46277)
0618 22:39:17.601274 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:17.601281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:17.627628 (+ 26347us) server_negotiation.cc:407] Connection header received
0618 22:39:17.630355 (+ 2727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:17.630363 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:17.630466 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:17.630530 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:17.634542 (+ 4012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.635671 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.637748 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.638355 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.638466 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.640989 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:17.640996 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:17.641000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:17.641051 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:17.644681 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.646500 (+ 1819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.646505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.646509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.646690 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.647288 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.647293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.647296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.647627 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:17.647748 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:17.648823 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0618 22:39:17.649052 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":111}
I20260618 22:39:17.650719 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.601385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:46277 (local address 127.0.0.1:43626)
0618 22:39:17.627511 (+ 26126us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:17.627532 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:17.628866 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:17.630773 (+ 1907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:17.630778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:17.631349 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:17.631679 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.631691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.635841 (+ 4150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.635846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.637624 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.637635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.638713 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.638719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.638986 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.639758 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:17.639777 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:17.640802 (+ 1025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:17.645940 (+ 5138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:17.645944 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:17.645947 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:17.646361 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:17.646822 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:17.646827 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:17.646831 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:17.647097 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:17.647842 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:17.647847 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:17.648503 (+ 656us) client_negotiation.cc:769] Sending connection context
0618 22:39:17.649983 (+ 1480us) client_negotiation.cc:240] Negotiation successful
0618 22:39:17.650281 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25987,"mutex_wait_us":68,"spinlock_wait_cycles":1408,"thread_start_us":19369,"threads_started":1}
I20260618 22:39:17.652460 12687 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b13db14e31f04ab08280cf6a17ff0810 (127.5.213.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } }
I20260618 22:39:17.652966 12741 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b13db14e31f04ab08280cf6a17ff0810 (127.5.213.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } }
I20260618 22:39:17.655864 12769 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } }
I20260618 22:39:17.656008 12769 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:39:17.658098 12880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } }
I20260618 22:39:17.658375 12880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:17.658957 12687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } }
I20260618 22:39:17.659176 12687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: This master's current role is: LEADER
Jun 18 22:39:17 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822354, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.254@KRBTEST.COM for oryx/127.5.213.252@KRBTEST.COM
I20260618 22:39:17.673527 12877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.658648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:37039 (local address 127.0.0.1:59510)
0618 22:39:17.658798 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:17.658818 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:17.658934 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:17.659528 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:17.659533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:17.659916 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:17.660074 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.660084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.661423 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.661428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.662343 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.662355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.663399 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.663405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.663593 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.664210 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:17.664232 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:17.666923 (+ 2691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:17.670863 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:17.670869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:17.670874 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:17.671399 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:17.671785 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:17.671790 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:17.671794 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:17.671945 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:17.672548 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:17.672553 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:17.672739 (+ 186us) client_negotiation.cc:769] Sending connection context
0618 22:39:17.672898 (+ 159us) client_negotiation.cc:240] Negotiation successful
0618 22:39:17.673169 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260618 22:39:17.673542 12850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.658736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.5.213.252:37039)
0618 22:39:17.658882 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:17.658888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:17.658906 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:39:17.659118 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:17.659123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:17.659241 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:17.659342 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:17.660220 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.661287 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.662478 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.663240 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.663371 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.667108 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:17.667115 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:17.667120 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:17.667161 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:17.670732 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.671520 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.671526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.671530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.671672 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.672093 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.672097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.672101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.672432 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:17.672562 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:17.672993 (+ 431us) server_negotiation.cc:299] Negotiation successful
0618 22:39:17.673222 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:39:17.675350 12127 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.213.252:37039: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0d65f54c7db549a8baf9dd615270b424"
member_type: NON_VOTER
last_known_addr {
host: "127.5.213.252"
port: 37039
}
attrs {
promote: true
}
}
dest_uuid: "798ea03554f649e8b59eff0c77e07c60"
cas_config_opid_index: 6
I20260618 22:39:17.676224 12127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: NON_VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: true } }
I20260618 22:39:17.679119 12822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:17.679375 12741 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 1 FOLLOWER]: Refusing update from remote peer 798ea03554f649e8b59eff0c77e07c60: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260618 22:39:17.679980 12875 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260618 22:39:17.682731 12822 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 1 FOLLOWER]: Refusing update from remote peer 798ea03554f649e8b59eff0c77e07c60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260618 22:39:17.683516 12875 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: NON_VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:17.684799 12741 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0d65f54c7db549a8baf9dd615270b424 (127.5.213.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: NON_VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: true } } }
I20260618 22:39:17.688359 12769 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: NON_VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: true } } }
I20260618 22:39:17.684369 12687 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0d65f54c7db549a8baf9dd615270b424 (127.5.213.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: NON_VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: true } } }
I20260618 22:39:17.688642 12769 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:39:17.691915 12687 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.213.252:37039
I20260618 22:39:17.692206 12880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: NON_VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: true } } }
I20260618 22:39:17.692476 12880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:17.692700 12684 master.cc:537] Master@127.5.213.252:37039 shutting down...
I20260618 22:39:17.692732 12880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: NON_VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: true } } }
I20260618 22:39:17.693190 12880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:17.704013 12822 raft_consensus.cc:858] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: NON_VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } }
W20260618 22:39:17.704306 12822 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: NON_VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } }
I20260618 22:39:17.717198 12822 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b13db14e31f04ab08280cf6a17ff0810 (127.5.213.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: NON_VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: true } } }
I20260618 22:39:17.722391 12884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: NON_VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: true } } }
I20260618 22:39:17.722749 12884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260618 22:39:17.722922 12884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: NON_VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: true } } }
I20260618 22:39:17.722894 12875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: NON_VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: true } } }
I20260618 22:39:17.723104 12884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260618 22:39:17.723166 12875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:17.723381 12854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 798ea03554f649e8b59eff0c77e07c60. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: NON_VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: true } } }
I20260618 22:39:17.723570 12854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [sys.catalog]: This master's current role is: NON_PARTICIPANT
W20260618 22:39:17.723855 12113 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260618 22:39:17.725541 12113 consensus_peers.cc:487] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 -> Peer 0d65f54c7db549a8baf9dd615270b424 (127.5.213.252:37039): Couldn't send request to peer 0d65f54c7db549a8baf9dd615270b424. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260618 22:39:17.726472 12684 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260618 22:39:17.726936 12684 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260618 22:39:17.727111 12684 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424: stopping tablet replica
I20260618 22:39:17.734071 12684 master.cc:552] Master@127.5.213.252:37039 shutdown complete.
I20260618 22:39:17.734133 12684 master_runner.cc:305] Clearing existing system tablet
I20260618 22:39:17.734428 12684 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:17.734503 12684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:17.736624 12684 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260618 22:39:17.737845 12890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:17.738175 12684 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:17.738392 12684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 22:39:17.738487 12684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/wal
uuid: "0d65f54c7db549a8baf9dd615270b424"
format_stamp: "Formatted at 2026-06-18 22:39:17 on dist-test-slave-lnj3"
server_key: "ebde671697b08444011b4c1830ccbfb0"
server_key_iv: "30b1dcfd412d9ebfc0bfa97c68b2582e"
server_key_version: "encryptionkey@0"
I20260618 22:39:17.741457 12684 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:17.745769 12684 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260618 22:39:17.745893 12684 log.cc:1196] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260618 22:39:17.746268 12684 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424: Deleting consensus metadata
I20260618 22:39:17.746464 12684 master_runner.cc:315] Copying system tablet from 127.5.213.254:33087
I20260618 22:39:17.748780 12684 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424: tablet copy: Beginning tablet copy session from remote peer at address 127.5.213.254:33087
I20260618 22:39:17.760521 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.749460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48400 (local address 127.5.213.254:33087)
0618 22:39:17.749708 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:17.749714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:17.750171 (+ 457us) server_negotiation.cc:407] Connection header received
0618 22:39:17.750341 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:17.750346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:17.750418 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:17.750498 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:17.751292 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.752061 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.753401 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.753807 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.753911 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.755605 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:17.755609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:17.755613 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:17.755643 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:17.757964 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.758540 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.758546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.758549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.758719 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:17.759051 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:17.759057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:17.759061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:17.759392 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:17.759514 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:17.759925 (+ 411us) server_negotiation.cc:299] Negotiation successful
0618 22:39:17.760156 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260618 22:39:17.760547 12896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:17.749422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.0.0.1:48400)
0618 22:39:17.750069 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:17.750086 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:17.750218 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:17.750537 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:17.750542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:17.750967 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:17.751153 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.751162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.752196 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.752202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.753277 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:17.753289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:17.753939 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:17.753944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:17.754102 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:17.754738 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:17.754755 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:17.755434 (+ 679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:17.758108 (+ 2674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:17.758113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:17.758117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:17.758436 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:17.758815 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:17.758820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:17.758825 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:17.758959 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:17.759492 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:17.759497 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:17.759654 (+ 157us) client_negotiation.cc:769] Sending connection context
0618 22:39:17.759828 (+ 174us) client_negotiation.cc:240] Negotiation successful
0618 22:39:17.760132 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":403,"threads_started":1}
I20260618 22:39:17.761323 12147 tablet_copy_service.cc:140] P 798ea03554f649e8b59eff0c77e07c60: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0d65f54c7db549a8baf9dd615270b424 ({username='oryx', principal='oryx/127.5.213.252@KRBTEST.COM'} at 127.0.0.1:48400)
I20260618 22:39:17.761464 12147 tablet_copy_service.cc:161] P 798ea03554f649e8b59eff0c77e07c60: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0d65f54c7db549a8baf9dd615270b424 at {username='oryx', principal='oryx/127.5.213.252@KRBTEST.COM'} at 127.0.0.1:48400: session id = 0d65f54c7db549a8baf9dd615270b424-00000000000000000000000000000000
I20260618 22:39:17.763712 12147 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60: Tablet Copy: opened 0 blocks and 1 log segments
I20260618 22:39:17.765141 12684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260618 22:39:17.769771 12684 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424: tablet copy: Starting download of 0 data blocks...
I20260618 22:39:17.769991 12684 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424: tablet copy: Starting download of 1 WAL segments...
I20260618 22:39:17.771852 12684 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260618 22:39:17.774677 12147 tablet_copy_service.cc:342] P 798ea03554f649e8b59eff0c77e07c60: Request end of tablet copy session 0d65f54c7db549a8baf9dd615270b424-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.213.252@KRBTEST.COM'} at 127.0.0.1:48400
I20260618 22:39:17.774791 12147 tablet_copy_service.cc:433] P 798ea03554f649e8b59eff0c77e07c60: ending tablet copy session 0d65f54c7db549a8baf9dd615270b424-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0d65f54c7db549a8baf9dd615270b424
I20260618 22:39:17.780192 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 12188
I20260618 22:39:17.784612 12684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:17.786940 12897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:17.792018 12898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:17.800978 12684 server_base.cc:1034] running on GCE node
W20260618 22:39:17.800899 12900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:17.801261 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:43029
--local_ip_for_outbound_sockets=127.5.213.193
--tserver_master_addrs=127.5.213.254:33087
--webserver_port=32841
--webserver_interface=127.5.213.193
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.193.keytab
--principal=oryx/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.213.254:33087,127.5.213.253:46277
--tserver_master_addrs=127.5.213.254:33087,127.5.213.253:46277,127.5.213.252:37039 with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:39:17.801537 12684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:17.802134 12684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:17.803313 12684 hybrid_clock.cc:648] HybridClock initialized: now 1781822357803261 us; error 43 us; skew 500 ppm
I20260618 22:39:17.803508 12684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:17 dist-test-slave-lnj3 krb5kdc[12082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822357, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:17.805186 12684 init.cc:377] Logged in from keytab as oryx/127.5.213.252@KRBTEST.COM (short username oryx)
I20260618 22:39:17.807431 12684 webserver.cc:460] Webserver started at http://127.5.213.252:42001/ using document root <none> and password file <none>
I20260618 22:39:17.807740 12684 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:17.807839 12684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:17.810209 12684 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260618 22:39:17.811694 12908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:17.812068 12684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 22:39:17.812175 12684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/wal
uuid: "0d65f54c7db549a8baf9dd615270b424"
format_stamp: "Formatted at 2026-06-18 22:39:17 on dist-test-slave-lnj3"
server_key: "ebde671697b08444011b4c1830ccbfb0"
server_key_iv: "30b1dcfd412d9ebfc0bfa97c68b2582e"
server_key_version: "encryptionkey@0"
I20260618 22:39:17.812351 12684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:17.831605 12684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:17.834864 12684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:17.856881 12684 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.252:37039
I20260618 22:39:17.856907 12959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.252:37039 every 8 connection(s)
I20260618 22:39:17.857728 12684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/master-2/data/info.pb
I20260618 22:39:17.863206 12960 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:39:17.867321 12960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424: Bootstrap starting.
I20260618 22:39:17.879160 12960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:39:17.879555 12960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424: Bootstrap complete.
I20260618 22:39:17.880345 12960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: NON_VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: true } }
I20260618 22:39:17.880548 12960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d65f54c7db549a8baf9dd615270b424, State: Initialized, Role: LEARNER
I20260618 22:39:17.880779 12960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: NON_VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: true } }
I20260618 22:39:17.881464 12963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: NON_VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: true } } }
I20260618 22:39:17.881727 12960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:39:17.881891 12963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [sys.catalog]: This master's current role is: LEARNER
I20260618 22:39:17.891959 12974 catalog_manager.cc:1259] Loaded cluster ID: 5d782181ba1a42da86921e7b22f586f2
I20260618 22:39:17.892047 12974 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424: loading cluster ID for follower catalog manager: success
I20260618 22:39:17.894321 12974 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424: acquiring CA information for follower catalog manager: success
I20260618 22:39:17.896118 12974 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260618 22:39:18.177819 12906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:18.178117 12906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:18.178246 12906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:18.185900 12906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:18.186012 12906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:18.186159 12906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:39:18.195977 12906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.193.keytab
--principal=oryx/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:43029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=32841
--tserver_master_addrs=127.5.213.254:33087,127.5.213.253:46277,127.5.213.252:37039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:18.197309 12906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:18.198663 12906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:18.210847 12980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:18.210917 12979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:18.210882 12982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:18.211966 12906 server_base.cc:1034] running on GCE node
I20260618 22:39:18.212596 12906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:18.213554 12906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:18.214715 12906 hybrid_clock.cc:648] HybridClock initialized: now 1781822358214658 us; error 43 us; skew 500 ppm
I20260618 22:39:18.215008 12906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:18 dist-test-slave-lnj3 krb5kdc[12082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822358, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:18.220546 12906 init.cc:377] Logged in from keytab as oryx/127.5.213.193@KRBTEST.COM (short username oryx)
I20260618 22:39:18.221854 12906 webserver.cc:460] Webserver started at http://127.5.213.193:32841/ using document root <none> and password file <none>
I20260618 22:39:18.222470 12906 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:18.222574 12906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:18.227912 12906 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:39:18.230350 12989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:18.231681 12906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:39:18.231822 12906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "ebff97d8cce5416a95db1cf751e0018b"
format_stamp: "Formatted at 2026-06-18 22:39:15 on dist-test-slave-lnj3"
server_key: "973d91ce162a5b6d0c719f21bb054cc0"
server_key_iv: "35e20c04cb6f4f928001099d727d6f44"
server_key_version: "encryptionkey@0"
I20260618 22:39:18.232684 12906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:18.248886 12906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:18.252532 12906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:18.252739 12906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:18.253595 12906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:18.254882 12906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:18.254966 12906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:18.255043 12906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:18.255093 12906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:18.283675 12906 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:43029
I20260618 22:39:18.283725 13101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:43029 every 8 connection(s)
I20260618 22:39:18.285024 12906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:39:18.294497 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 12906
I20260618 22:39:18.294852 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 12322
Jun 18 22:39:18 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822358, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.253@KRBTEST.COM
Jun 18 22:39:18 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822358, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM
Jun 18 22:39:18 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822358, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.252@KRBTEST.COM
I20260618 22:39:18.312490 13106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:18.287661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:34607 (local address 127.5.213.253:46277)
0618 22:39:18.289134 (+ 1473us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:18.289140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:18.293718 (+ 4578us) server_negotiation.cc:407] Connection header received
0618 22:39:18.293878 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:18.293884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:18.293970 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:18.294106 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:18.295647 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.296794 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.299615 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.300265 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.300378 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:18.305708 (+ 5330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:18.305715 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:18.305721 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:18.305764 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:18.308937 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:18.309645 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:18.309651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:18.309655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:18.309787 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:18.310379 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:18.310385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:18.310388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:18.310746 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:18.310888 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:18.311901 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0618 22:39:18.312117 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1360,"thread_start_us":1279,"threads_started":1}
I20260618 22:39:18.313755 13110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:18.288404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:46277 (local address 127.5.213.193:34607)
0618 22:39:18.293610 (+ 5206us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:18.293626 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:18.293743 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:18.294265 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:18.294269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:18.294957 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:18.295437 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:18.295464 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.296989 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.296995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:18.299427 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:18.299442 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.300687 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.300694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:18.301110 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:18.302126 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:18.302160 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:18.305431 (+ 3271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:18.309084 (+ 3653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:18.309093 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:18.309110 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:18.309511 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:18.309919 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:18.309924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:18.309928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:18.310248 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:18.310961 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:18.310969 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:18.311449 (+ 480us) client_negotiation.cc:769] Sending connection context
0618 22:39:18.311784 (+ 335us) client_negotiation.cc:240] Negotiation successful
0618 22:39:18.312316 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5074,"mutex_wait_us":955,"spinlock_wait_cycles":8960,"thread_start_us":3470,"threads_started":1}
I20260618 22:39:18.315490 13104 heartbeater.cc:347] Connected to a master server at 127.5.213.253:46277
I20260618 22:39:18.315791 13104 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:18.316195 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:18.287649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:59149 (local address 127.5.213.254:33087)
0618 22:39:18.288830 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:18.288837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:18.289395 (+ 558us) server_negotiation.cc:407] Connection header received
0618 22:39:18.292816 (+ 3421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:18.292821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:18.292906 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:18.292992 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:18.295717 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.296888 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.299611 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.300298 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.300393 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:18.309008 (+ 8615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:18.309016 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:18.309022 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:18.309069 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:18.312362 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:18.313206 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:18.313211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:18.313215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:18.313364 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:18.313812 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:18.313817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:18.313820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:18.314141 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:18.314748 (+ 607us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:18.315632 (+ 884us) server_negotiation.cc:299] Negotiation successful
0618 22:39:18.315846 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1033,"thread_start_us":1356,"threads_started":1}
I20260618 22:39:18.316669 13104 heartbeater.cc:511] Master 127.5.213.253:46277 requested a full tablet report, sending...
I20260618 22:39:18.317150 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:18.288403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.193:59149)
0618 22:39:18.289234 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:18.289282 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:18.292046 (+ 2764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:18.293519 (+ 1473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:18.293531 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:18.294456 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:18.295546 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:18.295560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.297092 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.297097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:18.299437 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:18.299454 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.300689 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.300695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:18.301117 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:18.303513 (+ 2396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:18.303538 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:18.308197 (+ 4659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:18.312500 (+ 4303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:18.312504 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:18.312508 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:18.313078 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:18.313496 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:18.313501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:18.313504 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:18.313680 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:18.314806 (+ 1126us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:18.314810 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:18.315081 (+ 271us) client_negotiation.cc:769] Sending connection context
0618 22:39:18.316440 (+ 1359us) client_negotiation.cc:240] Negotiation successful
0618 22:39:18.316739 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":149,"spinlock_wait_cycles":19840,"thread_start_us":564,"threads_started":1}
I20260618 22:39:18.317935 13103 heartbeater.cc:347] Connected to a master server at 127.5.213.254:33087
I20260618 22:39:18.318065 13103 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:18.318454 13103 heartbeater.cc:511] Master 127.5.213.254:33087 requested a full tablet report, sending...
I20260618 22:39:18.319070 12731 ts_manager.cc:194] Registered new tserver with Master: ebff97d8cce5416a95db1cf751e0018b (127.5.213.193:43029)
I20260618 22:39:18.319504 12127 ts_manager.cc:194] Re-registered known tserver with Master: ebff97d8cce5416a95db1cf751e0018b (127.5.213.193:43029)
I20260618 22:39:18.319584 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:18.288523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:37039 (local address 127.5.213.193:55657)
0618 22:39:18.292911 (+ 4388us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:18.292928 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:18.293041 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:18.293563 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:18.293566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:18.294425 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:18.295437 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:18.295464 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.304725 (+ 9261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.304731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:18.306358 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:18.306374 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.308418 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.308424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:18.308616 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:18.309930 (+ 1314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:18.309964 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:18.313270 (+ 3306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:18.316901 (+ 3631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:18.316906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:18.316909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:18.317439 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:18.317853 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:18.317858 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:18.317861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:18.318007 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:18.318690 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:18.318695 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:18.318844 (+ 149us) client_negotiation.cc:769] Sending connection context
0618 22:39:18.318969 (+ 125us) client_negotiation.cc:240] Negotiation successful
0618 22:39:18.319209 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4264,"mutex_wait_us":201,"spinlock_wait_cycles":25600,"thread_start_us":4804,"threads_started":1}
I20260618 22:39:18.320395 12127 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:59149
I20260618 22:39:18.320422 13109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:18.288876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:55657 (local address 127.5.213.252:37039)
0618 22:39:18.292676 (+ 3800us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:18.292683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:18.293022 (+ 339us) server_negotiation.cc:407] Connection header received
0618 22:39:18.293166 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:18.293173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:18.293258 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:18.293358 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:18.301627 (+ 8269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.303239 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.306553 (+ 3314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.307240 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.307357 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:18.313516 (+ 6159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:18.313522 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:18.313527 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:18.313565 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:18.316713 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:18.317572 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:18.317577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:18.317580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:18.317713 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:18.318139 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:18.318144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:18.318147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:18.318467 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:18.318588 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:18.319904 (+ 1316us) server_negotiation.cc:299] Negotiation successful
0618 22:39:18.320110 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3648,"thread_start_us":3543,"threads_started":1}
I20260618 22:39:18.322010 12880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: NON_VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: true } } }
I20260618 22:39:18.322341 12880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:18.322597 13102 heartbeater.cc:347] Connected to a master server at 127.5.213.252:37039
I20260618 22:39:18.322726 13102 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:18.323098 13102 heartbeater.cc:511] Master 127.5.213.252:37039 requested a full tablet report, sending...
I20260618 22:39:18.324736 12925 ts_manager.cc:194] Registered new tserver with Master: ebff97d8cce5416a95db1cf751e0018b (127.5.213.193:43029)
Jun 18 22:39:18 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.213.194@KRBTEST.COM
I20260618 22:39:18.330658 13109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:18.322598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59526 (local address 127.5.213.252:37039)
0618 22:39:18.323034 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:18.323040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:18.324687 (+ 1647us) server_negotiation.cc:407] Connection header received
0618 22:39:18.325653 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:18.325657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:18.325739 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:18.325828 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:18.326291 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.327201 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.328662 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.329802 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.329915 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:18.330104 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:18.330286 (+ 182us) server_negotiation.cc:299] Negotiation successful
0618 22:39:18.330403 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303}
I20260618 22:39:18.330700 13116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:18.322174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:37039 (local address 127.0.0.1:59526)
0618 22:39:18.324589 (+ 2415us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:18.324614 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:18.325493 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:18.325862 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:18.325867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:18.325892 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:18.326161 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:18.326170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.327334 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.327338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:18.328508 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:18.328520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.329963 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.329968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:18.330150 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:18.330178 (+ 28us) client_negotiation.cc:769] Sending connection context
0618 22:39:18.330256 (+ 78us) client_negotiation.cc:240] Negotiation successful
0618 22:39:18.330382 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2287,"thread_start_us":2150,"threads_started":1}
I20260618 22:39:18.331449 12935 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260618 22:39:18.331812 12963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 798ea03554f649e8b59eff0c77e07c60. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: NON_VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: true } } }
I20260618 22:39:18.332042 12963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [sys.catalog]: This master's current role is: LEARNER
I20260618 22:39:18.332536 13115 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60: attempting to promote NON_VOTER 0d65f54c7db549a8baf9dd615270b424 to VOTER
I20260618 22:39:18.333072 13115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: false } }
I20260618 22:39:18.334789 12935 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 1 LEARNER]: Refusing update from remote peer 798ea03554f649e8b59eff0c77e07c60: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260618 22:39:18.334810 12741 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 1 FOLLOWER]: Refusing update from remote peer 798ea03554f649e8b59eff0c77e07c60: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260618 22:39:18.335361 13114 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260618 22:39:18.335886 13115 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260618 22:39:18.337152 13113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:18.295660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.5.213.194:43543)
0618 22:39:18.308071 (+ 12411us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:18.308081 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:18.313698 (+ 5617us) server_negotiation.cc:407] Connection header received
0618 22:39:18.313775 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:18.313783 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:18.314700 (+ 917us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:18.317084 (+ 2384us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:18.317537 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.319132 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.325484 (+ 6352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.326377 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.326506 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:18.331141 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:18.331157 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:18.331175 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:18.331215 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:18.334096 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:18.334872 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:18.334880 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:18.334884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:18.335031 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:18.335460 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:18.335466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:18.335469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:18.335855 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:18.336159 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:18.336553 (+ 394us) server_negotiation.cc:299] Negotiation successful
0618 22:39:18.336789 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12202,"thread_start_us":12426,"threads_started":1}
I20260618 22:39:18.338109 13114 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0d65f54c7db549a8baf9dd615270b424 (127.5.213.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: false } } }
I20260618 22:39:18.338584 12445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:18.342048 13114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: false } } }
I20260618 22:39:18.342399 13114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:18.342661 13114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: false } } }
I20260618 22:39:18.342901 13114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:18.342607 12741 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0d65f54c7db549a8baf9dd615270b424 (127.5.213.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: false } } }
I20260618 22:39:18.348311 13115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: false } } }
I20260618 22:39:18.348652 13115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:18.349118 12934 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0d65f54c7db549a8baf9dd615270b424 (127.5.213.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: false } } }
I20260618 22:39:18.352624 12963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: false } } }
I20260618 22:39:18.352823 12963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:39:18.353718 13121 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "798ea03554f649e8b59eff0c77e07c60" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: false } } }
I20260618 22:39:18.353972 13121 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:39:18.429188 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 12322
I20260618 22:39:18.442127 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:43543
--local_ip_for_outbound_sockets=127.5.213.194
--tserver_master_addrs=127.5.213.254:33087
--webserver_port=40319
--webserver_interface=127.5.213.194
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.194.keytab
--principal=oryx/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.213.254:33087,127.5.213.253:46277
--tserver_master_addrs=127.5.213.254:33087,127.5.213.253:46277,127.5.213.252:37039 with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:18.750748 13122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:18.750999 13122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:18.751091 13122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:18.758265 13122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:18.758386 13122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:18.758530 13122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:39:18.768124 13122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.194.keytab
--principal=oryx/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:43543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=40319
--tserver_master_addrs=127.5.213.254:33087,127.5.213.253:46277,127.5.213.252:37039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:18.769439 13122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:18.770778 13122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:18.783257 13128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:18.783324 13130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:18.783303 13127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:18.784163 13122 server_base.cc:1034] running on GCE node
I20260618 22:39:18.784737 13122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:18.785624 13122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:18.786814 13122 hybrid_clock.cc:648] HybridClock initialized: now 1781822358786753 us; error 55 us; skew 500 ppm
I20260618 22:39:18.787106 13122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:18 dist-test-slave-lnj3 krb5kdc[12082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822358, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:18.790053 13122 init.cc:377] Logged in from keytab as oryx/127.5.213.194@KRBTEST.COM (short username oryx)
I20260618 22:39:18.791404 13122 webserver.cc:460] Webserver started at http://127.5.213.194:40319/ using document root <none> and password file <none>
I20260618 22:39:18.792049 13122 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:18.792176 13122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:18.797520 13122 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260618 22:39:18.800213 13137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:18.801510 13122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260618 22:39:18.801661 13122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "639c8bb1736b47da9fe60e4606c1f654"
format_stamp: "Formatted at 2026-06-18 22:39:15 on dist-test-slave-lnj3"
server_key: "034b008d35e5c9e453cce73814f5d74a"
server_key_iv: "63d31ab00135c94ff72bf52530f3531e"
server_key_version: "encryptionkey@0"
I20260618 22:39:18.802579 13122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:18.818535 13122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:18.822285 13122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:18.822520 13122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:18.823406 13122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:18.824592 13122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:18.824684 13122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:18.824782 13122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:18.824847 13122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:18.853468 13122 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:43543
I20260618 22:39:18.853533 13249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:43543 every 8 connection(s)
I20260618 22:39:18.854923 13122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:39:18.859638 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 13122
I20260618 22:39:18.859920 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 12455
Jun 18 22:39:18 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.213.195@KRBTEST.COM
Jun 18 22:39:18 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822358, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.252@KRBTEST.COM
Jun 18 22:39:18 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822358, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM
Jun 18 22:39:18 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822358, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.253@KRBTEST.COM
I20260618 22:39:18.895799 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:18.861379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.5.213.195:43255)
0618 22:39:18.869287 (+ 7908us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:18.869298 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:18.869330 (+ 32us) server_negotiation.cc:407] Connection header received
0618 22:39:18.874573 (+ 5243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:18.874583 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:18.874864 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:18.875010 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:18.877200 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.878705 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.880468 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.881221 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.881343 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:18.888196 (+ 6853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:18.888212 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:18.888230 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:18.888278 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:18.891936 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:18.892813 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:18.892822 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:18.892826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:18.892972 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:18.893409 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:18.893414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:18.893417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:18.893908 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:18.894172 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:18.895077 (+ 905us) server_negotiation.cc:299] Negotiation successful
0618 22:39:18.895336 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7717,"thread_start_us":7578,"threads_started":1}
I20260618 22:39:18.896562 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:18.858360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:51275 (local address 127.5.213.252:37039)
0618 22:39:18.873903 (+ 15543us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:18.873909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:18.878289 (+ 4380us) server_negotiation.cc:407] Connection header received
0618 22:39:18.878356 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:18.878362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:18.878457 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:18.878550 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:18.880816 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.881904 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.884183 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.884766 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.884854 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:18.889385 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:18.889391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:18.889395 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:18.889432 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:18.892993 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:18.893899 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:18.893905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:18.893909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:18.894036 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:18.894559 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:18.894563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:18.894567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:18.894915 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:18.895038 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:18.895971 (+ 933us) server_negotiation.cc:299] Negotiation successful
0618 22:39:18.896184 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15430,"thread_start_us":15539,"threads_started":1}
I20260618 22:39:18.897749 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:18.859027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:55693 (local address 127.5.213.254:33087)
0618 22:39:18.868083 (+ 9056us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:18.868090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:18.874035 (+ 5945us) server_negotiation.cc:407] Connection header received
0618 22:39:18.874104 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:18.874110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:18.874188 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:18.874302 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:18.881090 (+ 6788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.882145 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.885656 (+ 3511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.886300 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.886427 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:18.890908 (+ 4481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:18.890914 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:18.890920 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:18.890959 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:18.893948 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:18.894705 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:18.894710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:18.894713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:18.894857 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:18.895293 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:18.895299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:18.895302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:18.895597 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:18.895727 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:18.897226 (+ 1499us) server_negotiation.cc:299] Negotiation successful
0618 22:39:18.897436 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8918,"thread_start_us":8820,"threads_started":1}
I20260618 22:39:18.898114 12578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:18.898118 13254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:18.858107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:37039 (local address 127.5.213.194:51275)
0618 22:39:18.878015 (+ 19908us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:18.878038 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:18.878175 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:18.879019 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:18.879024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:18.879697 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:18.880642 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:18.880667 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.882078 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.882082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:18.884012 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:18.884026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.885189 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.885196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:18.885526 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:18.886245 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:18.886274 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:18.889134 (+ 2860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:18.893177 (+ 4043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:18.893185 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:18.893203 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:18.893771 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:18.894197 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:18.894201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:18.894204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:18.894437 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:18.895137 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:18.895149 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:18.895568 (+ 419us) client_negotiation.cc:769] Sending connection context
0618 22:39:18.896791 (+ 1223us) client_negotiation.cc:240] Negotiation successful
0618 22:39:18.897042 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19744,"mutex_wait_us":96,"spinlock_wait_cycles":59904,"thread_start_us":21966,"threads_started":1}
I20260618 22:39:18.898167 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:18.858720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:52613 (local address 127.5.213.253:46277)
0618 22:39:18.866836 (+ 8116us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:18.866844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:18.880633 (+ 13789us) server_negotiation.cc:407] Connection header received
0618 22:39:18.881644 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:18.881649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:18.881725 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:18.881824 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:18.883169 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.884180 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.885878 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.886405 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.886495 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:18.890877 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:18.890883 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:18.890888 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:18.890926 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:18.894446 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:18.895207 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:18.895212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:18.895215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:18.895341 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:18.895850 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:18.895855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:18.895858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:18.896165 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:18.896298 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:18.897344 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0618 22:39:18.897537 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7972,"thread_start_us":7886,"threads_started":1}
I20260618 22:39:18.898988 13255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:18.861955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:46277 (local address 127.5.213.194:52613)
0618 22:39:18.880521 (+ 18566us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:18.880540 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:18.880698 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:18.881996 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:18.882001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:18.882686 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:18.883016 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:18.883026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.884312 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.884317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:18.885721 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:18.885732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.886950 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.886957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:18.887147 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:18.887657 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:18.887677 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:18.890665 (+ 2988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:18.894619 (+ 3954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:18.894623 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:18.894659 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:18.895086 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:18.895478 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:18.895482 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:18.895485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:18.895729 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:18.896591 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:18.896606 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:18.897073 (+ 467us) client_negotiation.cc:769] Sending connection context
0618 22:39:18.898360 (+ 1287us) client_negotiation.cc:240] Negotiation successful
0618 22:39:18.898618 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18449,"mutex_wait_us":1,"thread_start_us":7570,"threads_started":1}
I20260618 22:39:18.899647 13250 heartbeater.cc:347] Connected to a master server at 127.5.213.252:37039
I20260618 22:39:18.899942 13250 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:18.900671 13250 heartbeater.cc:511] Master 127.5.213.252:37039 requested a full tablet report, sending...
I20260618 22:39:18.901427 13252 heartbeater.cc:347] Connected to a master server at 127.5.213.253:46277
I20260618 22:39:18.901576 13252 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:18.902006 13252 heartbeater.cc:511] Master 127.5.213.253:46277 requested a full tablet report, sending...
I20260618 22:39:18.902102 12925 ts_manager.cc:194] Registered new tserver with Master: 639c8bb1736b47da9fe60e4606c1f654 (127.5.213.194:43543)
I20260618 22:39:18.903223 12731 ts_manager.cc:194] Registered new tserver with Master: 639c8bb1736b47da9fe60e4606c1f654 (127.5.213.194:43543)
I20260618 22:39:18.903833 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:18.858107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.194:55693)
0618 22:39:18.870009 (+ 11902us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:18.870083 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:18.872871 (+ 2788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:18.878701 (+ 5830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:18.878715 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:18.879614 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:18.880936 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:18.880946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.882658 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.882664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:18.884222 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:18.884238 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:18.886687 (+ 2449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:18.886694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:18.886930 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:18.887477 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:18.887499 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:18.890675 (+ 3176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:18.894082 (+ 3407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:18.894086 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:18.894089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:18.894572 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:18.894991 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:18.894995 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:18.894998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:18.895156 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:18.895790 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:18.895795 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:18.895936 (+ 141us) client_negotiation.cc:769] Sending connection context
0618 22:39:18.896077 (+ 141us) client_negotiation.cc:240] Negotiation successful
0618 22:39:18.896632 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11543,"mutex_wait_us":3,"spinlock_wait_cycles":9088,"thread_start_us":20496,"threads_started":1}
I20260618 22:39:18.904781 13251 heartbeater.cc:347] Connected to a master server at 127.5.213.254:33087
I20260618 22:39:18.904887 13251 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:18.905184 13251 heartbeater.cc:511] Master 127.5.213.254:33087 requested a full tablet report, sending...
I20260618 22:39:18.905979 12127 ts_manager.cc:194] Re-registered known tserver with Master: 639c8bb1736b47da9fe60e4606c1f654 (127.5.213.194:43543)
I20260618 22:39:18.906854 12127 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:55693
I20260618 22:39:18.984076 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 12455
I20260618 22:39:18.996713 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:43255
--local_ip_for_outbound_sockets=127.5.213.195
--tserver_master_addrs=127.5.213.254:33087
--webserver_port=33847
--webserver_interface=127.5.213.195
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.195.keytab
--principal=oryx/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.213.254:33087,127.5.213.253:46277
--tserver_master_addrs=127.5.213.254:33087,127.5.213.253:46277,127.5.213.252:37039 with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:19.306033 13264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:19.306313 13264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:19.306434 13264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:19.313654 13264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:19.313781 13264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:19.313913 13264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:39:19.322793 13103 heartbeater.cc:503] Master 127.5.213.254:33087 was elected leader, sending a full tablet report...
I20260618 22:39:19.324380 13264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.195.keytab
--principal=oryx/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:43255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=33847
--tserver_master_addrs=127.5.213.254:33087,127.5.213.253:46277,127.5.213.252:37039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:19.325693 13264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:19.327061 13264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:19.339713 13269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:19.339658 13272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:19.340651 13270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:19.340814 13264 server_base.cc:1034] running on GCE node
I20260618 22:39:19.341521 13264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:19.342578 13264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:19.343806 13264 hybrid_clock.cc:648] HybridClock initialized: now 1781822359343744 us; error 54 us; skew 500 ppm
I20260618 22:39:19.344156 13264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:19 dist-test-slave-lnj3 krb5kdc[12082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822359, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:19.347883 13264 init.cc:377] Logged in from keytab as oryx/127.5.213.195@KRBTEST.COM (short username oryx)
I20260618 22:39:19.349483 13264 webserver.cc:460] Webserver started at http://127.5.213.195:33847/ using document root <none> and password file <none>
I20260618 22:39:19.350287 13264 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:19.350421 13264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:19.355703 13264 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260618 22:39:19.358131 13279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:19.359496 13264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260618 22:39:19.359619 13264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "b834563cb2c64ddeb15499cbd0c68a4c"
format_stamp: "Formatted at 2026-06-18 22:39:16 on dist-test-slave-lnj3"
server_key: "de28b4aec6603f619847b791ff1a2135"
server_key_iv: "b249bba5007881b37bfe8eac042bef04"
server_key_version: "encryptionkey@0"
I20260618 22:39:19.360479 13264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:19.402524 13264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:19.406494 13264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:19.406769 13264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:19.407678 13264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:19.409035 13264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:19.409116 13264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:19.409199 13264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:19.409252 13264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:19.438350 13264 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:43255
I20260618 22:39:19.438416 13391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:43255 every 8 connection(s)
I20260618 22:39:19.439738 13264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:39:19.444805 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 13264
Jun 18 22:39:19 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822359, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.253@KRBTEST.COM
Jun 18 22:39:19 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822359, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.252@KRBTEST.COM
Jun 18 22:39:19 dist-test-slave-lnj3 krb5kdc[12082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822359, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM
I20260618 22:39:19.505725 13397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.442955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:51155 (local address 127.5.213.253:46277)
0618 22:39:19.455529 (+ 12574us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:19.455537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:19.482343 (+ 26806us) server_negotiation.cc:407] Connection header received
0618 22:39:19.484273 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:19.484279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:19.484371 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:19.484452 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:19.487180 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.488301 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.491324 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.491986 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.492083 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.497716 (+ 5633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:19.497723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:19.497728 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:19.497771 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:19.501709 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:19.502696 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:19.502702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:19.502705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:19.502862 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:19.503439 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:19.503448 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:19.503451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:19.503882 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:19.504028 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:19.505060 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0618 22:39:19.505285 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12420,"thread_start_us":12595,"threads_started":1}
I20260618 22:39:19.507802 13401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.444085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:46277 (local address 127.5.213.195:51155)
0618 22:39:19.482237 (+ 38152us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:19.482255 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:19.483924 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:19.484917 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:19.484931 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:19.485933 (+ 1002us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:19.486952 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:19.486983 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.488551 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.488556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:19.491147 (+ 2591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:19.491163 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.492458 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.492467 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:19.492849 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.493649 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:19.493686 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:19.496766 (+ 3080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:19.501881 (+ 5115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:19.501891 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:19.501910 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:19.502515 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:19.503017 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:19.503022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:19.503025 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:19.503299 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:19.504127 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:19.504137 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:19.504600 (+ 463us) client_negotiation.cc:769] Sending connection context
0618 22:39:19.505945 (+ 1345us) client_negotiation.cc:240] Negotiation successful
0618 22:39:19.506473 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38041,"mutex_wait_us":260,"spinlock_wait_cycles":36608,"thread_start_us":36876,"threads_started":1}
I20260618 22:39:19.510170 13394 heartbeater.cc:347] Connected to a master server at 127.5.213.253:46277
I20260618 22:39:19.510478 13399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.443661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.195:50551)
0618 22:39:19.480545 (+ 36884us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:19.480613 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:19.484301 (+ 3688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:19.484917 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:19.484931 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:19.485933 (+ 1002us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:19.486952 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:19.486983 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.488534 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.488540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:19.491148 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:19.491163 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.492458 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.492467 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:19.495951 (+ 3484us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.497429 (+ 1478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:19.497456 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:19.501074 (+ 3618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:19.507253 (+ 6179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:19.507258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:19.507262 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:19.507850 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:19.508606 (+ 756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:19.508611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:19.508613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:19.508731 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:19.509327 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:19.509332 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:19.509494 (+ 162us) client_negotiation.cc:769] Sending connection context
0618 22:39:19.509655 (+ 161us) client_negotiation.cc:240] Negotiation successful
0618 22:39:19.509932 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36498,"mutex_wait_us":3453,"thread_start_us":38218,"threads_started":1}
I20260618 22:39:19.510551 13394 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:19.511523 13394 heartbeater.cc:511] Master 127.5.213.253:46277 requested a full tablet report, sending...
I20260618 22:39:19.512151 13398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.443003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:49199 (local address 127.5.213.252:37039)
0618 22:39:19.455915 (+ 12912us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:19.455919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:19.481311 (+ 25392us) server_negotiation.cc:407] Connection header received
0618 22:39:19.484224 (+ 2913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:19.484230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:19.484309 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:19.484409 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:19.487776 (+ 3367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.488925 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.492804 (+ 3879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.493883 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.494011 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.504358 (+ 10347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:19.504365 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:19.504370 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:19.504406 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:19.508828 (+ 4422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:19.509515 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:19.509522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:19.509525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:19.509684 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:19.510165 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:19.510170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:19.510173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:19.510842 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:19.510952 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:19.511390 (+ 438us) server_negotiation.cc:299] Negotiation successful
0618 22:39:19.511666 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12814,"thread_start_us":12736,"threads_started":1}
I20260618 22:39:19.513274 12731 ts_manager.cc:194] Registered new tserver with Master: b834563cb2c64ddeb15499cbd0c68a4c (127.5.213.195:43255)
I20260618 22:39:19.513345 13400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.443635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:37039 (local address 127.5.213.195:49199)
0618 22:39:19.481225 (+ 37590us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:19.481243 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:19.483923 (+ 2680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:19.484920 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:19.484931 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:19.485816 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:19.487042 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:19.487055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.489170 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.489175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:19.492362 (+ 3187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:19.492375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.494331 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.494341 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:19.494529 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.495174 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:19.495199 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:19.503278 (+ 8079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:19.508933 (+ 5655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:19.508938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:19.508941 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:19.509341 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:19.509825 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:19.509830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:19.509834 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:19.510021 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:19.511014 (+ 993us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:19.511017 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:19.511134 (+ 117us) client_negotiation.cc:769] Sending connection context
0618 22:39:19.512596 (+ 1462us) client_negotiation.cc:240] Negotiation successful
0618 22:39:19.512885 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37436,"mutex_wait_us":539,"spinlock_wait_cycles":75264,"thread_start_us":38407,"threads_started":1}
I20260618 22:39:19.514125 13392 heartbeater.cc:347] Connected to a master server at 127.5.213.252:37039
I20260618 22:39:19.514254 13392 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:19.514551 13410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.469072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59542 (local address 127.5.213.252:37039)
0618 22:39:19.479939 (+ 10867us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:19.479946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:19.481152 (+ 1206us) server_negotiation.cc:407] Connection header received
0618 22:39:19.481227 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:19.481233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:19.481310 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:19.481416 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:19.485127 (+ 3711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.486213 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.490157 (+ 3944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.490798 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.490908 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.496520 (+ 5612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:19.496527 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:19.496531 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:19.496567 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:19.501515 (+ 4948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:19.502388 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:19.502393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:19.502397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:19.502532 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:19.503807 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:19.503812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:19.503815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:19.504145 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:19.504758 (+ 613us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:19.513986 (+ 9228us) server_negotiation.cc:299] Negotiation successful
0618 22:39:19.514208 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10730,"thread_start_us":10607,"threads_started":1}
I20260618 22:39:19.514897 13396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.442672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:50551 (local address 127.5.213.254:33087)
0618 22:39:19.460713 (+ 18041us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:19.460719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:19.480763 (+ 20044us) server_negotiation.cc:407] Connection header received
0618 22:39:19.484459 (+ 3696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:19.484465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:19.484547 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:19.484631 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:19.487179 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.488301 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.491331 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.491980 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.492094 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.503519 (+ 11425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:19.503526 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:19.503531 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:19.503569 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:19.507142 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:19.508335 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:19.508340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:19.508343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:19.508487 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:19.508860 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:19.508865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:19.508868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:19.509201 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:19.509348 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:19.514316 (+ 4968us) server_negotiation.cc:299] Negotiation successful
0618 22:39:19.514536 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17879,"thread_start_us":446,"threads_started":1}
I20260618 22:39:19.514964 13392 heartbeater.cc:511] Master 127.5.213.252:37039 requested a full tablet report, sending...
I20260618 22:39:19.515724 13393 heartbeater.cc:347] Connected to a master server at 127.5.213.254:33087
I20260618 22:39:19.515838 13393 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:19.516232 13393 heartbeater.cc:511] Master 127.5.213.254:33087 requested a full tablet report, sending...
I20260618 22:39:19.516352 12924 ts_manager.cc:194] Registered new tserver with Master: b834563cb2c64ddeb15499cbd0c68a4c (127.5.213.195:43255)
I20260618 22:39:19.516968 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.468219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.5.213.253:46277)
0618 22:39:19.491441 (+ 23222us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:19.491447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:19.492687 (+ 1240us) server_negotiation.cc:407] Connection header received
0618 22:39:19.492735 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:19.492740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:19.492806 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:19.492891 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:19.498658 (+ 5767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.499832 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.502964 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.503655 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.503758 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.507626 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:19.507632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:19.507638 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:19.507673 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:19.511301 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:19.512715 (+ 1414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:19.512720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:19.512723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:19.512882 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:19.515115 (+ 2233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:19.515119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:19.515123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:19.515450 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:19.515674 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:19.516210 (+ 536us) server_negotiation.cc:299] Negotiation successful
0618 22:39:19.516566 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23063,"thread_start_us":24355,"threads_started":1}
I20260618 22:39:19.517135 12127 ts_manager.cc:194] Re-registered known tserver with Master: b834563cb2c64ddeb15499cbd0c68a4c (127.5.213.195:43255)
I20260618 22:39:19.518234 12127 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:50551
I20260618 22:39:19.520712 13408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.468669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.5.213.254:33087)
0618 22:39:19.492358 (+ 23689us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:19.492363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:19.492379 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:39:19.492421 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:19.492425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:19.492494 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:19.495535 (+ 3041us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:19.495709 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.496771 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.499266 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.499893 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.499991 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.503000 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:19.503005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:19.503010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:19.503045 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:19.518040 (+ 14995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:19.518819 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:19.518824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:19.518828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:19.518981 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:19.519402 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:19.519406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:19.519410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:19.519711 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:19.519944 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:19.520168 (+ 224us) server_negotiation.cc:299] Negotiation successful
0618 22:39:19.520366 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23581,"thread_start_us":23977,"threads_started":1}
I20260618 22:39:19.523937 12127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:39:19.530606 12127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:39:19.558805 13427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.549369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:43543 (local address 127.0.0.1:49480)
0618 22:39:19.550008 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:19.550025 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:19.550120 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:19.551814 (+ 1694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:19.551818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:19.551843 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:19.552198 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:19.552209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.554086 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.554091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:19.556219 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:19.556234 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.558064 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.558068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:19.558193 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.558218 (+ 25us) client_negotiation.cc:769] Sending connection context
0618 22:39:19.558295 (+ 77us) client_negotiation.cc:240] Negotiation successful
0618 22:39:19.558410 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":23,"thread_start_us":570,"threads_started":1}
I20260618 22:39:19.558869 13428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.549824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49480 (local address 127.5.213.194:43543)
0618 22:39:19.551235 (+ 1411us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:19.551243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:19.551270 (+ 27us) server_negotiation.cc:407] Connection header received
0618 22:39:19.551351 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:19.551358 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:19.551649 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:19.551822 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:19.552360 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.553945 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.556392 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.557890 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.558034 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.558268 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:19.558342 (+ 74us) server_negotiation.cc:299] Negotiation successful
0618 22:39:19.558533 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1270,"thread_start_us":1169,"threads_started":1}
I20260618 22:39:19.562445 13185 tablet_service.cc:1467] Processing CreateTablet for tablet 20a3ffb6529940168ea465643c852117 (DEFAULT_TABLE table=test-table [id=4ba486f2b0af407aa148ba508b8c6dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:19.564046 13185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a3ffb6529940168ea465643c852117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:19.564231 13425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.548071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:43029 (local address 127.0.0.1:47518)
0618 22:39:19.550710 (+ 2639us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:19.550727 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:19.550836 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:19.551558 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:19.551564 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:19.551595 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:19.552020 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:19.552033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.553932 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.553937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:19.556219 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:19.556234 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.563429 (+ 7195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.563435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:19.563602 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.563636 (+ 34us) client_negotiation.cc:769] Sending connection context
0618 22:39:19.563734 (+ 98us) client_negotiation.cc:240] Negotiation successful
0618 22:39:19.563880 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":42,"thread_start_us":1086,"threads_started":1}
I20260618 22:39:19.564459 13426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.548278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47518 (local address 127.5.213.193:43029)
0618 22:39:19.548814 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:19.548824 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:19.550821 (+ 1997us) server_negotiation.cc:407] Connection header received
0618 22:39:19.550980 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:19.550989 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:19.551296 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:19.551537 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:19.552235 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.553813 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.560035 (+ 6222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.561615 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.561754 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.561962 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:19.563811 (+ 1849us) server_negotiation.cc:299] Negotiation successful
0618 22:39:19.564044 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":246,"threads_started":1}
I20260618 22:39:19.567734 13037 tablet_service.cc:1467] Processing CreateTablet for tablet 20a3ffb6529940168ea465643c852117 (DEFAULT_TABLE table=test-table [id=4ba486f2b0af407aa148ba508b8c6dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:19.568943 13037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a3ffb6529940168ea465643c852117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:19.572095 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.547541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:43255 (local address 127.0.0.1:47670)
0618 22:39:19.548078 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:19.548096 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:19.548210 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:19.564256 (+ 16046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:19.564262 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:19.564300 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:19.564597 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:19.564609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.567114 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.567120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:19.569068 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:19.569083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.571357 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.571363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:19.571538 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.571572 (+ 34us) client_negotiation.cc:769] Sending connection context
0618 22:39:19.571653 (+ 81us) client_negotiation.cc:240] Negotiation successful
0618 22:39:19.571781 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":308,"threads_started":1}
I20260618 22:39:19.572198 13424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.547778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47670 (local address 127.5.213.195:43255)
0618 22:39:19.563152 (+ 15374us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:19.563161 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:19.563204 (+ 43us) server_negotiation.cc:407] Connection header received
0618 22:39:19.563297 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:19.563304 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:19.563536 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:19.563727 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:19.565417 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.566956 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.569284 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.570676 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.570789 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.570981 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:19.571732 (+ 751us) server_negotiation.cc:299] Negotiation successful
0618 22:39:19.571898 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15227,"thread_start_us":15054,"threads_started":1}
I20260618 22:39:19.575677 13327 tablet_service.cc:1467] Processing CreateTablet for tablet 20a3ffb6529940168ea465643c852117 (DEFAULT_TABLE table=test-table [id=4ba486f2b0af407aa148ba508b8c6dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:19.577001 13327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a3ffb6529940168ea465643c852117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:19.580848 13430 tablet_bootstrap.cc:492] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b: Bootstrap starting.
I20260618 22:39:19.583335 13429 tablet_bootstrap.cc:492] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654: Bootstrap starting.
I20260618 22:39:19.584043 13430 tablet_bootstrap.cc:654] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:19.585242 13430 log.cc:824] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:19.586406 13429 tablet_bootstrap.cc:654] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:19.587838 13429 log.cc:824] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:19.588342 13430 tablet_bootstrap.cc:492] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b: No bootstrap required, opened a new log
I20260618 22:39:19.588609 13430 ts_tablet_manager.cc:1397] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260618 22:39:19.590129 13429 tablet_bootstrap.cc:492] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654: No bootstrap required, opened a new log
I20260618 22:39:19.590358 13429 ts_tablet_manager.cc:1397] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260618 22:39:19.591753 13433 tablet_bootstrap.cc:492] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c: Bootstrap starting.
I20260618 22:39:19.595094 13433 tablet_bootstrap.cc:654] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:19.596288 13433 log.cc:824] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:19.596674 13430 raft_consensus.cc:348] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 } } peers { permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 } } peers { permanent_uuid: "639c8bb1736b47da9fe60e4606c1f654" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 43543 } }
I20260618 22:39:19.597101 13430 raft_consensus.cc:374] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:19.597206 13430 raft_consensus.cc:729] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebff97d8cce5416a95db1cf751e0018b, State: Initialized, Role: FOLLOWER
I20260618 22:39:19.597743 13430 consensus_queue.cc:260] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 } } peers { permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 } } peers { permanent_uuid: "639c8bb1736b47da9fe60e4606c1f654" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 43543 } }
I20260618 22:39:19.599231 13430 ts_tablet_manager.cc:1428] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260618 22:39:19.600390 13433 tablet_bootstrap.cc:492] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c: No bootstrap required, opened a new log
I20260618 22:39:19.600255 13429 raft_consensus.cc:348] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 } } peers { permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 } } peers { permanent_uuid: "639c8bb1736b47da9fe60e4606c1f654" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 43543 } }
I20260618 22:39:19.600647 13433 ts_tablet_manager.cc:1397] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260618 22:39:19.600638 13429 raft_consensus.cc:374] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:19.600735 13429 raft_consensus.cc:729] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 639c8bb1736b47da9fe60e4606c1f654, State: Initialized, Role: FOLLOWER
I20260618 22:39:19.601524 13429 consensus_queue.cc:260] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 } } peers { permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 } } peers { permanent_uuid: "639c8bb1736b47da9fe60e4606c1f654" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 43543 } }
I20260618 22:39:19.607347 13251 heartbeater.cc:503] Master 127.5.213.254:33087 was elected leader, sending a full tablet report...
I20260618 22:39:19.608024 13429 ts_tablet_manager.cc:1428] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654: Time spent starting tablet: real 0.018s user 0.013s sys 0.005s
I20260618 22:39:19.609566 13436 raft_consensus.cc:482] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:39:19.610257 13436 raft_consensus.cc:504] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 } } peers { permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 } } peers { permanent_uuid: "639c8bb1736b47da9fe60e4606c1f654" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 43543 } }
I20260618 22:39:19.618408 13436 leader_election.cc:288] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b834563cb2c64ddeb15499cbd0c68a4c (127.5.213.195:43255), ebff97d8cce5416a95db1cf751e0018b (127.5.213.193:43029)
I20260618 22:39:19.619508 13433 raft_consensus.cc:348] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 } } peers { permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 } } peers { permanent_uuid: "639c8bb1736b47da9fe60e4606c1f654" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 43543 } }
I20260618 22:39:19.619910 13433 raft_consensus.cc:374] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:19.620012 13433 raft_consensus.cc:729] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b834563cb2c64ddeb15499cbd0c68a4c, State: Initialized, Role: FOLLOWER
I20260618 22:39:19.620692 13433 consensus_queue.cc:260] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 } } peers { permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 } } peers { permanent_uuid: "639c8bb1736b47da9fe60e4606c1f654" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 43543 } }
I20260618 22:39:19.623911 13433 ts_tablet_manager.cc:1428] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c: Time spent starting tablet: real 0.023s user 0.015s sys 0.000s
I20260618 22:39:19.624223 13393 heartbeater.cc:503] Master 127.5.213.254:33087 was elected leader, sending a full tablet report...
I20260618 22:39:19.627573 13424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.612603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:36407 (local address 127.5.213.195:43255)
0618 22:39:19.612788 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:19.612795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:19.619252 (+ 6457us) server_negotiation.cc:407] Connection header received
0618 22:39:19.619518 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:19.619523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:19.619595 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:19.619654 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:19.620748 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.621760 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.625045 (+ 3285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.626194 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.626314 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.626385 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:19.627213 (+ 828us) server_negotiation.cc:299] Negotiation successful
0618 22:39:19.627341 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 22:39:19.628433 13439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.612129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:43255 (local address 127.5.213.194:36407)
0618 22:39:19.619129 (+ 7000us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:19.619151 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:19.619360 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:19.619955 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:19.619959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:19.620026 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:19.620570 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:19.620584 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.621990 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.621994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:19.624878 (+ 2884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:19.624891 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.626748 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.626755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:19.626915 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.626946 (+ 31us) client_negotiation.cc:769] Sending connection context
0618 22:39:19.627938 (+ 992us) client_negotiation.cc:240] Negotiation successful
0618 22:39:19.628063 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6857,"mutex_wait_us":146,"thread_start_us":6972,"threads_started":1}
I20260618 22:39:19.628870 13426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.612801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:43923 (local address 127.5.213.193:43029)
0618 22:39:19.612979 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:19.612984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:19.619424 (+ 6440us) server_negotiation.cc:407] Connection header received
0618 22:39:19.619866 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:19.619871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:19.619948 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:19.620038 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:19.620770 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.621889 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.626230 (+ 4341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.627777 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.627905 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.627972 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:19.628510 (+ 538us) server_negotiation.cc:299] Negotiation successful
0618 22:39:19.628632 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 22:39:19.628921 13438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.612298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:43029 (local address 127.5.213.194:43923)
0618 22:39:19.619341 (+ 7043us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:19.619356 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:19.619712 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:19.620190 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:19.620194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:19.620217 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:19.620579 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:19.620592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.622086 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.622092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:19.625483 (+ 3391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:19.625496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.628170 (+ 2674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.628176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:19.628361 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.628388 (+ 27us) client_negotiation.cc:769] Sending connection context
0618 22:39:19.628470 (+ 82us) client_negotiation.cc:240] Negotiation successful
0618 22:39:19.628600 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6942,"thread_start_us":6493,"threads_started":1}
I20260618 22:39:19.629473 13347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a3ffb6529940168ea465643c852117" candidate_uuid: "639c8bb1736b47da9fe60e4606c1f654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" is_pre_election: true
I20260618 22:39:19.629858 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a3ffb6529940168ea465643c852117" candidate_uuid: "639c8bb1736b47da9fe60e4606c1f654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebff97d8cce5416a95db1cf751e0018b" is_pre_election: true
I20260618 22:39:19.629990 13347 raft_consensus.cc:2454] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 639c8bb1736b47da9fe60e4606c1f654 in term 0.
I20260618 22:39:19.630329 13057 raft_consensus.cc:2454] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 639c8bb1736b47da9fe60e4606c1f654 in term 0.
I20260618 22:39:19.630754 13140 leader_election.cc:302] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 639c8bb1736b47da9fe60e4606c1f654, b834563cb2c64ddeb15499cbd0c68a4c; no voters:
I20260618 22:39:19.631167 13437 raft_consensus.cc:2781] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:39:19.631290 13437 raft_consensus.cc:482] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:39:19.631426 13437 raft_consensus.cc:3037] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:19.633672 13437 raft_consensus.cc:504] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 } } peers { permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 } } peers { permanent_uuid: "639c8bb1736b47da9fe60e4606c1f654" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 43543 } }
I20260618 22:39:19.634231 13437 leader_election.cc:288] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [CANDIDATE]: Term 1 election: Requested vote from peers b834563cb2c64ddeb15499cbd0c68a4c (127.5.213.195:43255), ebff97d8cce5416a95db1cf751e0018b (127.5.213.193:43029)
I20260618 22:39:19.634589 13347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a3ffb6529940168ea465643c852117" candidate_uuid: "639c8bb1736b47da9fe60e4606c1f654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b834563cb2c64ddeb15499cbd0c68a4c"
I20260618 22:39:19.634615 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a3ffb6529940168ea465643c852117" candidate_uuid: "639c8bb1736b47da9fe60e4606c1f654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebff97d8cce5416a95db1cf751e0018b"
I20260618 22:39:19.634778 13347 raft_consensus.cc:3037] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:19.634837 13057 raft_consensus.cc:3037] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:19.637121 13057 raft_consensus.cc:2454] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 639c8bb1736b47da9fe60e4606c1f654 in term 1.
I20260618 22:39:19.637253 13347 raft_consensus.cc:2454] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 639c8bb1736b47da9fe60e4606c1f654 in term 1.
I20260618 22:39:19.637514 13138 leader_election.cc:302] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 639c8bb1736b47da9fe60e4606c1f654, ebff97d8cce5416a95db1cf751e0018b; no voters:
I20260618 22:39:19.637795 13437 raft_consensus.cc:2781] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:19.637965 13437 raft_consensus.cc:686] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [term 1 LEADER]: Becoming Leader. State: Replica: 639c8bb1736b47da9fe60e4606c1f654, State: Running, Role: LEADER
I20260618 22:39:19.638435 13437 consensus_queue.cc:237] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 } } peers { permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 } } peers { permanent_uuid: "639c8bb1736b47da9fe60e4606c1f654" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 43543 } }
I20260618 22:39:19.643137 12127 catalog_manager.cc:5495] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 reported cstate change: term changed from 0 to 1, leader changed from <none> to 639c8bb1736b47da9fe60e4606c1f654 (127.5.213.194). New cstate: current_term: 1 leader_uuid: "639c8bb1736b47da9fe60e4606c1f654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "639c8bb1736b47da9fe60e4606c1f654" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 43543 } health_report { overall_health: HEALTHY } } }
I20260618 22:39:19.681258 13428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.676650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49496 (local address 127.5.213.194:43543)
0618 22:39:19.676806 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:19.676810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:19.676825 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:39:19.676905 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:19.676909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:19.676980 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:19.677103 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:39:19.677361 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.678041 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.679310 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.679697 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.679786 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.680256 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:39:19.680761 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:39:19.680881 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:19.680923 (+ 42us) server_negotiation.cc:299] Negotiation successful
0618 22:39:19.680985 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:39:19.688874 13347 raft_consensus.cc:1261] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c [term 1 FOLLOWER]: Refusing update from remote peer 639c8bb1736b47da9fe60e4606c1f654: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:39:19.688880 13057 raft_consensus.cc:1261] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b [term 1 FOLLOWER]: Refusing update from remote peer 639c8bb1736b47da9fe60e4606c1f654: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:39:19.689764 13436 consensus_queue.cc:1035] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:19.690026 13437 consensus_queue.cc:1035] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260618 22:39:19.692605 13395 tablet.cc:2369] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:39:19.703079 13443 mvcc.cc:205] Tried to move back new op lower bound from 7298344385276588032 to 7298344385082834944. Current Snapshot: MvccSnapshot[applied={T|T < 7298344385276588032}]
I20260618 22:39:19.706350 13444 mvcc.cc:205] Tried to move back new op lower bound from 7298344385276588032 to 7298344385082834944. Current Snapshot: MvccSnapshot[applied={T|T < 7298344385276588032}]
I20260618 22:39:19.724967 12127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:39:19.725244 12127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:39:19.731979 12127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 798ea03554f649e8b59eff0c77e07c60: Sending DeleteTablet for 3 replicas of tablet 20a3ffb6529940168ea465643c852117
I20260618 22:39:19.733006 13327 tablet_service.cc:1514] Processing DeleteTablet for tablet 20a3ffb6529940168ea465643c852117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:19 UTC) from {username='oryx', principal='oryx/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:47670
I20260618 22:39:19.733141 13037 tablet_service.cc:1514] Processing DeleteTablet for tablet 20a3ffb6529940168ea465643c852117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:19 UTC) from {username='oryx', principal='oryx/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:47518
I20260618 22:39:19.735736 13453 tablet_replica.cc:330] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b: stopping tablet replica
I20260618 22:39:19.735644 13185 tablet_service.cc:1514] Processing DeleteTablet for tablet 20a3ffb6529940168ea465643c852117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:19 UTC) from {username='oryx', principal='oryx/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:49480
I20260618 22:39:19.736135 13453 raft_consensus.cc:2229] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:39:19.736611 13453 raft_consensus.cc:2258] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:19.737311 13452 tablet_replica.cc:330] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c: stopping tablet replica
I20260618 22:39:19.737675 13452 raft_consensus.cc:2229] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:39:19.738014 13452 raft_consensus.cc:2258] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:19.740147 13453 ts_tablet_manager.cc:1905] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:19.740166 13452 ts_tablet_manager.cc:1905] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:19.742179 13455 tablet_replica.cc:330] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654: stopping tablet replica
I20260618 22:39:19.742532 13455 raft_consensus.cc:2229] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:39:19.743196 13455 raft_consensus.cc:2258] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:19.746150 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 12095
I20260618 22:39:19.747128 12157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:19.747195 13453 ts_tablet_manager.cc:1918] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:39:19.747208 13455 ts_tablet_manager.cc:1905] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:19.747340 13453 log.cc:1196] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/20a3ffb6529940168ea465643c852117
I20260618 22:39:19.747848 13453 ts_tablet_manager.cc:1939] T 20a3ffb6529940168ea465643c852117 P ebff97d8cce5416a95db1cf751e0018b: Deleting consensus metadata
I20260618 22:39:19.753307 13452 ts_tablet_manager.cc:1918] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:39:19.753476 13452 log.cc:1196] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/20a3ffb6529940168ea465643c852117
I20260618 22:39:19.753996 13452 ts_tablet_manager.cc:1939] T 20a3ffb6529940168ea465643c852117 P b834563cb2c64ddeb15499cbd0c68a4c: Deleting consensus metadata
I20260618 22:39:19.762158 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.736352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43640 (local address 127.5.213.253:46277)
0618 22:39:19.736521 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:19.736527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:19.745309 (+ 8782us) server_negotiation.cc:407] Connection header received
0618 22:39:19.745518 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:19.745524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:19.745634 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:19.745727 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:19.748493 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.749557 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.751539 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.752206 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.752327 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.755862 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:19.755869 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:19.755874 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:19.755913 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:19.759224 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:19.760105 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:19.760111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:19.760115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:19.760251 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:19.760621 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:19.760626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:19.760629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:19.760958 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:19.761191 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:19.761538 (+ 347us) server_negotiation.cc:299] Negotiation successful
0618 22:39:19.761749 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 22:39:19.767324 13455 ts_tablet_manager.cc:1918] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:39:19.767491 13455 log.cc:1196] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/20a3ffb6529940168ea465643c852117
I20260618 22:39:19.768080 13455 ts_tablet_manager.cc:1939] T 20a3ffb6529940168ea465643c852117 P 639c8bb1736b47da9fe60e4606c1f654: Deleting consensus metadata
I20260618 22:39:19.778213 13410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.745529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59552 (local address 127.5.213.252:37039)
0618 22:39:19.745966 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:19.745993 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:19.762116 (+ 16123us) server_negotiation.cc:407] Connection header received
0618 22:39:19.762298 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:19.762304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:19.762398 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:19.762489 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:19.764426 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.765500 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.767939 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.768608 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.768704 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.772039 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:19.772046 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:19.772050 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:19.772101 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:19.775368 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:19.776177 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:19.776183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:19.776186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:19.776358 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:19.776753 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:19.776758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:19.776761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:19.777083 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:19.777310 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:19.777652 (+ 342us) server_negotiation.cc:299] Negotiation successful
0618 22:39:19.777864 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312}
I20260618 22:39:19.891592 12113 catalog_manager.cc:4841] TS b834563cb2c64ddeb15499cbd0c68a4c (127.5.213.195:43255): tablet 20a3ffb6529940168ea465643c852117 (table test-table [id=4ba486f2b0af407aa148ba508b8c6dec]) successfully deleted
W20260618 22:39:19.891901 12113 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.5.213.195:43255, user_credentials={real_user=slave}} blocked reactor thread for 136203us
I20260618 22:39:19.892298 12113 catalog_manager.cc:4841] TS 639c8bb1736b47da9fe60e4606c1f654 (127.5.213.194:43543): tablet 20a3ffb6529940168ea465643c852117 (table test-table [id=4ba486f2b0af407aa148ba508b8c6dec]) successfully deleted
I20260618 22:39:19.892786 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 12095
I20260618 22:39:19.938138 13410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.922777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58862 (local address 127.5.213.252:37039)
0618 22:39:19.922960 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:19.922967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:19.924099 (+ 1132us) server_negotiation.cc:407] Connection header received
0618 22:39:19.924263 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:19.924276 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:19.924350 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:19.924431 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:19.926540 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.927646 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.929439 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.929906 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.930016 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.932474 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:19.932479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:19.932482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:19.932513 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:19.935017 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:19.935780 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:19.935785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:19.935789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:19.935921 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:19.936336 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:19.936341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:19.936344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:19.936680 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:19.936987 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:19.937565 (+ 578us) server_negotiation.cc:299] Negotiation successful
0618 22:39:19.937775 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 22:39:19.939316 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:19.924036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33306 (local address 127.5.213.253:46277)
0618 22:39:19.924210 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:19.924216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:19.924764 (+ 548us) server_negotiation.cc:407] Connection header received
0618 22:39:19.924892 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:19.924897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:19.924981 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:19.925086 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:19.926958 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.927637 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.928966 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:19.929376 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:19.929507 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:19.933041 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:19.933047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:19.933051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:19.933102 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:19.936562 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:19.937097 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:19.937103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:19.937107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:19.937237 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:19.937560 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:19.937565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:19.937568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:19.937821 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:19.938019 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:19.938711 (+ 692us) server_negotiation.cc:299] Negotiation successful
0618 22:39:19.939023 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 22:39:20.709462 13463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:20.708730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.194:59985)
0618 22:39:20.709134 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:20.709276 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":196,"threads_started":1}
W20260618 22:39:20.709811 13251 heartbeater.cc:644] Failed to heartbeat to 127.5.213.254:33087 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
I20260618 22:39:20.716161 13464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:20.715042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.195:34117)
0618 22:39:20.715525 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:20.715669 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":273,"threads_started":1}
I20260618 22:39:20.716270 13465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:20.715539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.193:45707)
0618 22:39:20.715982 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:20.716104 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":225,"threads_started":1}
W20260618 22:39:20.716532 13393 heartbeater.cc:644] Failed to heartbeat to 127.5.213.254:33087 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
W20260618 22:39:20.716650 13103 heartbeater.cc:644] Failed to heartbeat to 127.5.213.254:33087 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
I20260618 22:39:21.231632 13467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 798ea03554f649e8b59eff0c77e07c60)
I20260618 22:39:21.231711 13466 raft_consensus.cc:482] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 798ea03554f649e8b59eff0c77e07c60)
I20260618 22:39:21.231873 13467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: false } }
I20260618 22:39:21.231962 13466 raft_consensus.cc:504] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: false } }
I20260618 22:39:21.233193 13466 leader_election.cc:288] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 798ea03554f649e8b59eff0c77e07c60 (127.5.213.254:33087), 0d65f54c7db549a8baf9dd615270b424 (127.5.213.252:37039)
I20260618 22:39:21.233457 13467 leader_election.cc:288] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 798ea03554f649e8b59eff0c77e07c60 (127.5.213.254:33087), b13db14e31f04ab08280cf6a17ff0810 (127.5.213.253:46277)
I20260618 22:39:21.234356 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.233290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.0.0.1:52328)
0618 22:39:21.233931 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.234113 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"threads_started":1}
I20260618 22:39:21.235772 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.233325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.0.0.1:52344)
0618 22:39:21.235292 (+ 1967us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.235501 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1854,"threads_started":1}
W20260618 22:39:21.242244 12716 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
W20260618 22:39:21.244534 12910 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
I20260618 22:39:21.244968 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.233671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:37039 (local address 127.0.0.1:58872)
0618 22:39:21.236550 (+ 2879us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.236566 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:21.236688 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:21.237220 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:21.237225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:21.237251 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:21.237479 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:21.237490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.239084 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.239090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:21.241355 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:21.241371 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.243991 (+ 2620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.243999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:21.244257 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:21.244299 (+ 42us) client_negotiation.cc:769] Sending connection context
0618 22:39:21.244405 (+ 106us) client_negotiation.cc:240] Negotiation successful
0618 22:39:21.244579 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2785,"thread_start_us":1371,"threads_started":1}
I20260618 22:39:21.245800 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.234159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.5.213.252:37039)
0618 22:39:21.236383 (+ 2224us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:21.236390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:21.236790 (+ 400us) server_negotiation.cc:407] Connection header received
0618 22:39:21.236857 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:21.236862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:21.236930 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:21.237009 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:21.237632 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.238908 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.242107 (+ 3199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.243606 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.243759 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:21.243881 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:21.245191 (+ 1310us) server_negotiation.cc:299] Negotiation successful
0618 22:39:21.245349 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2095,"thread_start_us":1825,"threads_started":1}
I20260618 22:39:21.246417 12934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b13db14e31f04ab08280cf6a17ff0810" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0d65f54c7db549a8baf9dd615270b424" is_pre_election: true
I20260618 22:39:21.246814 12934 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b13db14e31f04ab08280cf6a17ff0810 in term 1.
I20260618 22:39:21.247486 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.247100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.0.0.1:52350)
0618 22:39:21.247237 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.247322 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260618 22:39:21.247514 12717 leader_election.cc:302] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d65f54c7db549a8baf9dd615270b424, b13db14e31f04ab08280cf6a17ff0810; no voters:
W20260618 22:39:21.247777 12716 leader_election.cc:334] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 798ea03554f649e8b59eff0c77e07c60 (127.5.213.254:33087): Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
I20260618 22:39:21.247907 13466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260618 22:39:21.248020 13466 raft_consensus.cc:482] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 798ea03554f649e8b59eff0c77e07c60)
I20260618 22:39:21.248144 13466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 1 FOLLOWER]: Advancing to term 2
I20260618 22:39:21.248991 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.233295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:46277 (local address 127.0.0.1:33310)
0618 22:39:21.234618 (+ 1323us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.234655 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:21.234794 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:21.242255 (+ 7461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:21.242264 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:21.242296 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:21.242655 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:21.242667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.244356 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.244361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:21.246448 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:21.246464 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.248314 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.248321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:21.248530 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:21.248572 (+ 42us) client_negotiation.cc:769] Sending connection context
0618 22:39:21.248638 (+ 66us) client_negotiation.cc:240] Negotiation successful
0618 22:39:21.248745 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1237,"mutex_wait_us":191,"thread_start_us":259,"threads_started":1}
I20260618 22:39:21.249740 13471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.233590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.5.213.253:46277)
0618 22:39:21.241914 (+ 8324us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:21.241920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:21.241941 (+ 21us) server_negotiation.cc:407] Connection header received
0618 22:39:21.242004 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:21.242009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:21.242097 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:21.242379 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:21.242984 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.244194 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.246668 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.248087 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.249198 (+ 1111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:21.249310 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:21.249359 (+ 49us) server_negotiation.cc:299] Negotiation successful
0618 22:39:21.249477 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8200,"thread_start_us":8013,"threads_started":1}
I20260618 22:39:21.249769 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.249369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.0.0.1:52366)
0618 22:39:21.249534 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.249621 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260618 22:39:21.250012 12910 leader_election.cc:334] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 798ea03554f649e8b59eff0c77e07c60 (127.5.213.254:33087): Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
I20260618 22:39:21.250350 12741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d65f54c7db549a8baf9dd615270b424" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b13db14e31f04ab08280cf6a17ff0810" is_pre_election: true
I20260618 22:39:21.251101 13466 raft_consensus.cc:504] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: false } }
I20260618 22:39:21.251507 12741 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d65f54c7db549a8baf9dd615270b424 in current term 2: Already voted for candidate b13db14e31f04ab08280cf6a17ff0810 in this term.
I20260618 22:39:21.251926 13466 leader_election.cc:288] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [CANDIDATE]: Term 2 election: Requested vote from peers 798ea03554f649e8b59eff0c77e07c60 (127.5.213.254:33087), 0d65f54c7db549a8baf9dd615270b424 (127.5.213.252:37039)
I20260618 22:39:21.252238 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.251878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.0.0.1:52370)
0618 22:39:21.252070 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.252148 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260618 22:39:21.252243 12934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b13db14e31f04ab08280cf6a17ff0810" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0d65f54c7db549a8baf9dd615270b424"
I20260618 22:39:21.252378 12912 leader_election.cc:302] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d65f54c7db549a8baf9dd615270b424; no voters: 798ea03554f649e8b59eff0c77e07c60, b13db14e31f04ab08280cf6a17ff0810
I20260618 22:39:21.252481 12934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 1 FOLLOWER]: Advancing to term 2
I20260618 22:39:21.253088 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.252791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.0.0.1:52384)
0618 22:39:21.252924 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.252981 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260618 22:39:21.253261 12716 leader_election.cc:334] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 798ea03554f649e8b59eff0c77e07c60 (127.5.213.254:33087): Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
I20260618 22:39:21.254736 12934 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b13db14e31f04ab08280cf6a17ff0810 in term 2.
I20260618 22:39:21.254916 13467 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:39:21.255115 12717 leader_election.cc:302] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d65f54c7db549a8baf9dd615270b424, b13db14e31f04ab08280cf6a17ff0810; no voters: 798ea03554f649e8b59eff0c77e07c60
I20260618 22:39:21.255379 13466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 2 FOLLOWER]: Leader election won for term 2
I20260618 22:39:21.255796 13466 raft_consensus.cc:686] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [term 2 LEADER]: Becoming Leader. State: Replica: b13db14e31f04ab08280cf6a17ff0810, State: Running, Role: LEADER
I20260618 22:39:21.256253 13466 consensus_queue.cc:237] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: false } }
I20260618 22:39:21.258292 13476 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b13db14e31f04ab08280cf6a17ff0810. Latest consensus state: current_term: 2 leader_uuid: "b13db14e31f04ab08280cf6a17ff0810" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: false } } }
I20260618 22:39:21.258816 13476 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:21.259408 13478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:39:21.262734 13478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:39:21.263468 13478 catalog_manager.cc:1259] Loaded cluster ID: 5d782181ba1a42da86921e7b22f586f2
I20260618 22:39:21.263535 13478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:39:21.264600 13478 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:39:21.265324 13478 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: Loaded TSK: 0
I20260618 22:39:21.266106 13478 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:39:21.313576 12731 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:39:21.320442 12731 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:39:21.324358 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.324021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.0.0.1:52406)
0618 22:39:21.324164 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.324236 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260618 22:39:21.324457 12934 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [term 2 FOLLOWER]: Refusing update from remote peer b13db14e31f04ab08280cf6a17ff0810: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260618 22:39:21.325060 13466 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260618 22:39:21.325335 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.324907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.0.0.1:52412)
0618 22:39:21.325055 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.325177 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260618 22:39:21.325484 12716 consensus_peers.cc:487] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 -> Peer 798ea03554f649e8b59eff0c77e07c60 (127.5.213.254:33087): Couldn't send request to peer 798ea03554f649e8b59eff0c77e07c60. Status: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260618 22:39:21.327402 13467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b13db14e31f04ab08280cf6a17ff0810. Latest consensus state: current_term: 2 leader_uuid: "b13db14e31f04ab08280cf6a17ff0810" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: false } } }
I20260618 22:39:21.327615 13467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:39:21.329028 13466 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b13db14e31f04ab08280cf6a17ff0810" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: false } } }
I20260618 22:39:21.329312 13466 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:21.329867 13480 mvcc.cc:205] Tried to move back new op lower bound from 7298344391978971136 to 7298344391709650944. Current Snapshot: MvccSnapshot[applied={T|T < 7298344391978971136}]
I20260618 22:39:21.329769 13467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b13db14e31f04ab08280cf6a17ff0810" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "798ea03554f649e8b59eff0c77e07c60" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33087 } } peers { permanent_uuid: "b13db14e31f04ab08280cf6a17ff0810" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 46277 } attrs { promote: false } } peers { permanent_uuid: "0d65f54c7db549a8baf9dd615270b424" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 37039 } attrs { promote: false } } }
I20260618 22:39:21.330168 13467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d65f54c7db549a8baf9dd615270b424 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:39:21.359839 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.337468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:43255 (local address 127.0.0.1:51992)
0618 22:39:21.337842 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.337860 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:21.337970 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:21.350808 (+ 12838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:21.350815 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:21.350850 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:21.351251 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:21.351270 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.353743 (+ 2473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.353749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:21.357534 (+ 3785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:21.357544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.359089 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.359096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:21.359280 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:21.359316 (+ 36us) client_negotiation.cc:769] Sending connection context
0618 22:39:21.359376 (+ 60us) client_negotiation.cc:240] Negotiation successful
0618 22:39:21.359509 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":1631}
I20260618 22:39:21.360728 13485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.337446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38994 (local address 127.5.213.193:43029)
0618 22:39:21.352809 (+ 15363us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:21.352815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:21.352835 (+ 20us) server_negotiation.cc:407] Connection header received
0618 22:39:21.352904 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:21.352909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:21.352990 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:21.353567 (+ 577us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:21.353719 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.354862 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.357748 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.359068 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.360141 (+ 1073us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:21.360256 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:21.360306 (+ 50us) server_negotiation.cc:299] Negotiation successful
0618 22:39:21.360438 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15228,"thread_start_us":15637,"threads_started":1}
I20260618 22:39:21.360029 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.337171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:43029 (local address 127.0.0.1:38994)
0618 22:39:21.337344 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.337365 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:21.337473 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:21.353233 (+ 15760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:21.353238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:21.353265 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:21.353441 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:21.353451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.355014 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.355020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:21.356944 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:21.356959 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.359257 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.359264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:21.359468 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:21.359504 (+ 36us) client_negotiation.cc:769] Sending connection context
0618 22:39:21.359565 (+ 61us) client_negotiation.cc:240] Negotiation successful
0618 22:39:21.359701 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":1}
I20260618 22:39:21.362039 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.338174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51992 (local address 127.5.213.195:43255)
0618 22:39:21.338598 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:21.338613 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:21.338651 (+ 38us) server_negotiation.cc:407] Connection header received
0618 22:39:21.338719 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:21.338725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:21.338811 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:21.350770 (+ 11959us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:21.351457 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.353600 (+ 2143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.357676 (+ 4076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.358877 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.361553 (+ 2676us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:21.361655 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:21.361703 (+ 48us) server_negotiation.cc:299] Negotiation successful
0618 22:39:21.361829 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":238,"threads_started":1}
I20260618 22:39:21.362185 13037 tablet_service.cc:1467] Processing CreateTablet for tablet 4812e968ce8a4d44af7d9fa2a3d4a027 (DEFAULT_TABLE table=test-table [id=e3bdebbcba764ff399b03596652ee1d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:21.362710 13037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4812e968ce8a4d44af7d9fa2a3d4a027. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:21.365083 13327 tablet_service.cc:1467] Processing CreateTablet for tablet 4812e968ce8a4d44af7d9fa2a3d4a027 (DEFAULT_TABLE table=test-table [id=e3bdebbcba764ff399b03596652ee1d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:21.365476 13327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4812e968ce8a4d44af7d9fa2a3d4a027. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:21.368535 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.339662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39814 (local address 127.5.213.194:43543)
0618 22:39:21.363643 (+ 23981us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:21.363649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:21.363668 (+ 19us) server_negotiation.cc:407] Connection header received
0618 22:39:21.363731 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:21.363737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:21.363801 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:21.363896 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:21.364223 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.365233 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.366852 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.367674 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.367785 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:21.367879 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:21.368121 (+ 242us) server_negotiation.cc:299] Negotiation successful
0618 22:39:21.368247 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23827,"thread_start_us":24325,"threads_started":1}
I20260618 22:39:21.368530 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.339253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:43543 (local address 127.0.0.1:39814)
0618 22:39:21.361214 (+ 21961us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.361230 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:21.361319 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:21.363923 (+ 2604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:21.363926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:21.363947 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:21.364074 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:21.364082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.365373 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.365378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:21.366737 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:21.366749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.367804 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.367810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:21.367982 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:21.368005 (+ 23us) client_negotiation.cc:769] Sending connection context
0618 22:39:21.368081 (+ 76us) client_negotiation.cc:240] Negotiation successful
0618 22:39:21.368239 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21819,"thread_start_us":25062,"threads_started":1}
I20260618 22:39:21.369400 13489 tablet_bootstrap.cc:492] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b: Bootstrap starting.
I20260618 22:39:21.370743 13489 tablet_bootstrap.cc:654] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:21.371310 13185 tablet_service.cc:1467] Processing CreateTablet for tablet 4812e968ce8a4d44af7d9fa2a3d4a027 (DEFAULT_TABLE table=test-table [id=e3bdebbcba764ff399b03596652ee1d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:21.371690 13185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4812e968ce8a4d44af7d9fa2a3d4a027. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:21.373134 13489 tablet_bootstrap.cc:492] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b: No bootstrap required, opened a new log
I20260618 22:39:21.373258 13489 ts_tablet_manager.cc:1397] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260618 22:39:21.373813 13489 raft_consensus.cc:348] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 } } peers { permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 } } peers { permanent_uuid: "639c8bb1736b47da9fe60e4606c1f654" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 43543 } }
I20260618 22:39:21.373994 13489 raft_consensus.cc:374] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:21.374079 13489 raft_consensus.cc:729] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebff97d8cce5416a95db1cf751e0018b, State: Initialized, Role: FOLLOWER
I20260618 22:39:21.374306 13489 consensus_queue.cc:260] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 } } peers { permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 } } peers { permanent_uuid: "639c8bb1736b47da9fe60e4606c1f654" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 43543 } }
I20260618 22:39:21.374918 13490 tablet_bootstrap.cc:492] T 4812e968ce8a4d44af7d9fa2a3d4a027 P b834563cb2c64ddeb15499cbd0c68a4c: Bootstrap starting.
I20260618 22:39:21.375025 13489 ts_tablet_manager.cc:1428] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260618 22:39:21.376677 13490 tablet_bootstrap.cc:654] T 4812e968ce8a4d44af7d9fa2a3d4a027 P b834563cb2c64ddeb15499cbd0c68a4c: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:21.380345 13494 tablet_bootstrap.cc:492] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654: Bootstrap starting.
I20260618 22:39:21.382212 13494 tablet_bootstrap.cc:654] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:21.382776 13104 heartbeater.cc:503] Master 127.5.213.253:46277 was elected leader, sending a full tablet report...
I20260618 22:39:21.386348 13493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.375558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.193:34597)
0618 22:39:21.385898 (+ 10340us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.386064 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10201,"thread_start_us":10035,"threads_started":1}
I20260618 22:39:21.386951 13490 tablet_bootstrap.cc:492] T 4812e968ce8a4d44af7d9fa2a3d4a027 P b834563cb2c64ddeb15499cbd0c68a4c: No bootstrap required, opened a new log
I20260618 22:39:21.387100 13490 ts_tablet_manager.cc:1397] T 4812e968ce8a4d44af7d9fa2a3d4a027 P b834563cb2c64ddeb15499cbd0c68a4c: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.006s
I20260618 22:39:21.387580 13493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.387135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.193:46827)
0618 22:39:21.387367 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.387447 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260618 22:39:21.388087 13494 tablet_bootstrap.cc:492] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654: No bootstrap required, opened a new log
I20260618 22:39:21.388024 13490 raft_consensus.cc:348] T 4812e968ce8a4d44af7d9fa2a3d4a027 P b834563cb2c64ddeb15499cbd0c68a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 } } peers { permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 } } peers { permanent_uuid: "639c8bb1736b47da9fe60e4606c1f654" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 43543 } }
I20260618 22:39:21.388213 13494 ts_tablet_manager.cc:1397] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260618 22:39:21.388223 13490 raft_consensus.cc:374] T 4812e968ce8a4d44af7d9fa2a3d4a027 P b834563cb2c64ddeb15499cbd0c68a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:21.388301 13490 raft_consensus.cc:729] T 4812e968ce8a4d44af7d9fa2a3d4a027 P b834563cb2c64ddeb15499cbd0c68a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b834563cb2c64ddeb15499cbd0c68a4c, State: Initialized, Role: FOLLOWER
I20260618 22:39:21.388516 13490 consensus_queue.cc:260] T 4812e968ce8a4d44af7d9fa2a3d4a027 P b834563cb2c64ddeb15499cbd0c68a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 } } peers { permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 } } peers { permanent_uuid: "639c8bb1736b47da9fe60e4606c1f654" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 43543 } }
I20260618 22:39:21.388844 13494 raft_consensus.cc:348] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 } } peers { permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 } } peers { permanent_uuid: "639c8bb1736b47da9fe60e4606c1f654" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 43543 } }
I20260618 22:39:21.389072 13494 raft_consensus.cc:374] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:21.389150 13490 ts_tablet_manager.cc:1428] T 4812e968ce8a4d44af7d9fa2a3d4a027 P b834563cb2c64ddeb15499cbd0c68a4c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260618 22:39:21.389163 13494 raft_consensus.cc:729] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 639c8bb1736b47da9fe60e4606c1f654, State: Initialized, Role: FOLLOWER
I20260618 22:39:21.389361 13494 consensus_queue.cc:260] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 } } peers { permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 } } peers { permanent_uuid: "639c8bb1736b47da9fe60e4606c1f654" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 43543 } }
I20260618 22:39:21.391651 13394 heartbeater.cc:503] Master 127.5.213.253:46277 was elected leader, sending a full tablet report...
I20260618 22:39:21.392817 13494 ts_tablet_manager.cc:1428] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20260618 22:39:21.393416 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.389902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.195:55461)
0618 22:39:21.393021 (+ 3119us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.393187 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2996,"thread_start_us":2861,"threads_started":1}
I20260618 22:39:21.395300 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.394081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.194:39709)
0618 22:39:21.394554 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.394703 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":257,"threads_started":1}
I20260618 22:39:21.395457 13252 heartbeater.cc:503] Master 127.5.213.253:46277 was elected leader, sending a full tablet report...
I20260618 22:39:21.396082 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.394019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.195:35427)
0618 22:39:21.395888 (+ 1869us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.395966 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1775}
I20260618 22:39:21.396420 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.396035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.194:36161)
0618 22:39:21.396210 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.396298 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260618 22:39:21.539424 13105 tablet.cc:2369] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:39:21.608393 13253 tablet.cc:2369] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:39:21.729944 13498 raft_consensus.cc:482] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:39:21.730121 13498 raft_consensus.cc:504] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 } } peers { permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 } } peers { permanent_uuid: "639c8bb1736b47da9fe60e4606c1f654" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 43543 } }
I20260618 22:39:21.730762 13498 leader_election.cc:288] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebff97d8cce5416a95db1cf751e0018b (127.5.213.193:43029), b834563cb2c64ddeb15499cbd0c68a4c (127.5.213.195:43255)
I20260618 22:39:21.731251 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4812e968ce8a4d44af7d9fa2a3d4a027" candidate_uuid: "639c8bb1736b47da9fe60e4606c1f654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebff97d8cce5416a95db1cf751e0018b" is_pre_election: true
I20260618 22:39:21.731292 13347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4812e968ce8a4d44af7d9fa2a3d4a027" candidate_uuid: "639c8bb1736b47da9fe60e4606c1f654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" is_pre_election: true
I20260618 22:39:21.731462 13057 raft_consensus.cc:2454] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 639c8bb1736b47da9fe60e4606c1f654 in term 0.
I20260618 22:39:21.731515 13347 raft_consensus.cc:2454] T 4812e968ce8a4d44af7d9fa2a3d4a027 P b834563cb2c64ddeb15499cbd0c68a4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 639c8bb1736b47da9fe60e4606c1f654 in term 0.
I20260618 22:39:21.731905 13138 leader_election.cc:302] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 639c8bb1736b47da9fe60e4606c1f654, ebff97d8cce5416a95db1cf751e0018b; no voters:
I20260618 22:39:21.732161 13498 raft_consensus.cc:2781] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:39:21.732263 13498 raft_consensus.cc:482] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:39:21.732331 13498 raft_consensus.cc:3037] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:21.734259 13498 raft_consensus.cc:504] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 } } peers { permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 } } peers { permanent_uuid: "639c8bb1736b47da9fe60e4606c1f654" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 43543 } }
I20260618 22:39:21.734754 13498 leader_election.cc:288] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [CANDIDATE]: Term 1 election: Requested vote from peers ebff97d8cce5416a95db1cf751e0018b (127.5.213.193:43029), b834563cb2c64ddeb15499cbd0c68a4c (127.5.213.195:43255)
I20260618 22:39:21.735226 13347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4812e968ce8a4d44af7d9fa2a3d4a027" candidate_uuid: "639c8bb1736b47da9fe60e4606c1f654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b834563cb2c64ddeb15499cbd0c68a4c"
I20260618 22:39:21.735227 13057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4812e968ce8a4d44af7d9fa2a3d4a027" candidate_uuid: "639c8bb1736b47da9fe60e4606c1f654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebff97d8cce5416a95db1cf751e0018b"
I20260618 22:39:21.735390 13057 raft_consensus.cc:3037] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:21.735401 13347 raft_consensus.cc:3037] T 4812e968ce8a4d44af7d9fa2a3d4a027 P b834563cb2c64ddeb15499cbd0c68a4c [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:21.737476 13347 raft_consensus.cc:2454] T 4812e968ce8a4d44af7d9fa2a3d4a027 P b834563cb2c64ddeb15499cbd0c68a4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 639c8bb1736b47da9fe60e4606c1f654 in term 1.
I20260618 22:39:21.737613 13057 raft_consensus.cc:2454] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 639c8bb1736b47da9fe60e4606c1f654 in term 1.
I20260618 22:39:21.737871 13140 leader_election.cc:302] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 639c8bb1736b47da9fe60e4606c1f654, b834563cb2c64ddeb15499cbd0c68a4c; no voters:
I20260618 22:39:21.738173 13498 raft_consensus.cc:2781] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:21.738651 13498 raft_consensus.cc:686] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [term 1 LEADER]: Becoming Leader. State: Replica: 639c8bb1736b47da9fe60e4606c1f654, State: Running, Role: LEADER
I20260618 22:39:21.738931 13498 consensus_queue.cc:237] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 } } peers { permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 } } peers { permanent_uuid: "639c8bb1736b47da9fe60e4606c1f654" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 43543 } }
I20260618 22:39:21.740087 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.739571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.194:41087)
0618 22:39:21.739758 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.739892 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260618 22:39:21.744045 12731 catalog_manager.cc:5495] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 reported cstate change: term changed from 0 to 1, leader changed from <none> to 639c8bb1736b47da9fe60e4606c1f654 (127.5.213.194). New cstate: current_term: 1 leader_uuid: "639c8bb1736b47da9fe60e4606c1f654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "639c8bb1736b47da9fe60e4606c1f654" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 43543 } health_report { overall_health: HEALTHY } } }
I20260618 22:39:21.809948 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.805676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39818 (local address 127.5.213.194:43543)
0618 22:39:21.805871 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:21.805877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:21.805895 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:39:21.806053 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:21.806059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:21.806125 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:21.806196 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:39:21.806544 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.807252 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.808457 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.808851 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.808954 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:21.809320 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:39:21.809416 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:39:21.809484 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:21.809618 (+ 134us) server_negotiation.cc:299] Negotiation successful
0618 22:39:21.809687 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:39:21.812711 13057 raft_consensus.cc:1261] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b [term 1 FOLLOWER]: Refusing update from remote peer 639c8bb1736b47da9fe60e4606c1f654: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:39:21.812721 13347 raft_consensus.cc:1261] T 4812e968ce8a4d44af7d9fa2a3d4a027 P b834563cb2c64ddeb15499cbd0c68a4c [term 1 FOLLOWER]: Refusing update from remote peer 639c8bb1736b47da9fe60e4606c1f654: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:39:21.813277 13501 consensus_queue.cc:1035] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b834563cb2c64ddeb15499cbd0c68a4c" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:21.813552 13498 consensus_queue.cc:1035] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebff97d8cce5416a95db1cf751e0018b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 43029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:21.815820 13493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.815274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.193:60079)
0618 22:39:21.815501 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.815621 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260618 22:39:21.816418 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.815461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.195:50505)
0618 22:39:21.816072 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.816196 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485}
I20260618 22:39:21.821410 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.820887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.194:41011)
0618 22:39:21.821149 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.821240 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260618 22:39:21.826071 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.825662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.195:60195)
0618 22:39:21.825809 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.825900 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260618 22:39:21.826862 12730 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:39:21.827111 12730 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:39:21.832417 12730 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b13db14e31f04ab08280cf6a17ff0810: Sending DeleteTablet for 3 replicas of tablet 4812e968ce8a4d44af7d9fa2a3d4a027
I20260618 22:39:21.833372 13037 tablet_service.cc:1514] Processing DeleteTablet for tablet 4812e968ce8a4d44af7d9fa2a3d4a027 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:21 UTC) from {username='oryx', principal='oryx/127.5.213.253@KRBTEST.COM'} at 127.0.0.1:38994
I20260618 22:39:21.834030 13511 tablet_replica.cc:330] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b: stopping tablet replica
I20260618 22:39:21.834362 13327 tablet_service.cc:1514] Processing DeleteTablet for tablet 4812e968ce8a4d44af7d9fa2a3d4a027 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:21 UTC) from {username='oryx', principal='oryx/127.5.213.253@KRBTEST.COM'} at 127.0.0.1:51992
I20260618 22:39:21.834571 13185 tablet_service.cc:1514] Processing DeleteTablet for tablet 4812e968ce8a4d44af7d9fa2a3d4a027 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:21 UTC) from {username='oryx', principal='oryx/127.5.213.253@KRBTEST.COM'} at 127.0.0.1:39814
I20260618 22:39:21.834290 13511 raft_consensus.cc:2229] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:39:21.835399 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 12906
I20260618 22:39:21.835602 13511 pending_rounds.cc:62] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b: Trying to abort 2 pending ops.
I20260618 22:39:21.840582 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.840072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.5.213.194:36219)
0618 22:39:21.840284 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.840371 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260618 22:39:21.841310 13138 consensus_peers.cc:487] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 -> Peer ebff97d8cce5416a95db1cf751e0018b (127.5.213.193:43029): Couldn't send request to peer ebff97d8cce5416a95db1cf751e0018b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260618 22:39:21.835738 13511 pending_rounds.cc:68] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7298344393979207680 op_type: WRITE_OP write_request { tablet_id: "4812e968ce8a4d44af7d9fa2a3d4a027" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\305\351\321\321\006\"8\n\ntest-admin\022*\n e3bdebbcba764ff399b03596652ee1d6\020\001\030\001 \001(\001" signature: "<redacted>""U\270\220*\241i`\301\247\2315\363/Jfw\006\007w7E9+r\021\253_\347\365\202\354{\367\234e\200\261\320\304\243\021\0219u|\276\261\331\nh\316T\223\007Z\3072ewL\372\363\236\311" signing_key_seq_num: 0 } } request_id { client_id: "e1c77c1d431e43058e6d8eb60af211b1" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260618 22:39:21.843472 13511 pending_rounds.cc:68] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7298344393684504576 op_type: NO_OP noop_request { }
I20260618 22:39:21.843597 13511 raft_consensus.cc:2866] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260618 22:39:21.843739 13511 raft_consensus.cc:2258] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:21.845427 13511 ts_tablet_manager.cc:1905] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:21.845535 13513 tablet_replica.cc:330] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654: stopping tablet replica
I20260618 22:39:21.845711 13513 raft_consensus.cc:2229] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:39:21.846171 13513 raft_consensus.cc:2258] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:21.846293 13512 tablet_replica.cc:330] T 4812e968ce8a4d44af7d9fa2a3d4a027 P b834563cb2c64ddeb15499cbd0c68a4c: stopping tablet replica
I20260618 22:39:21.846499 13512 raft_consensus.cc:2229] T 4812e968ce8a4d44af7d9fa2a3d4a027 P b834563cb2c64ddeb15499cbd0c68a4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:39:21.846841 13512 raft_consensus.cc:2258] T 4812e968ce8a4d44af7d9fa2a3d4a027 P b834563cb2c64ddeb15499cbd0c68a4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:21.847236 13513 ts_tablet_manager.cc:1905] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:21.847875 13512 ts_tablet_manager.cc:1905] T 4812e968ce8a4d44af7d9fa2a3d4a027 P b834563cb2c64ddeb15499cbd0c68a4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:21.852416 13511 ts_tablet_manager.cc:1918] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:39:21.852595 13511 log.cc:1196] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/4812e968ce8a4d44af7d9fa2a3d4a027
I20260618 22:39:21.853111 13513 ts_tablet_manager.cc:1918] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:39:21.853268 13513 log.cc:1196] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/4812e968ce8a4d44af7d9fa2a3d4a027
I20260618 22:39:21.853652 13511 ts_tablet_manager.cc:1939] T 4812e968ce8a4d44af7d9fa2a3d4a027 P ebff97d8cce5416a95db1cf751e0018b: Deleting consensus metadata
I20260618 22:39:21.853694 13513 ts_tablet_manager.cc:1939] T 4812e968ce8a4d44af7d9fa2a3d4a027 P 639c8bb1736b47da9fe60e4606c1f654: Deleting consensus metadata
I20260618 22:39:21.854780 12715 catalog_manager.cc:4841] TS ebff97d8cce5416a95db1cf751e0018b (127.5.213.193:43029): tablet 4812e968ce8a4d44af7d9fa2a3d4a027 (table test-table [id=e3bdebbcba764ff399b03596652ee1d6]) successfully deleted
I20260618 22:39:21.855084 12717 catalog_manager.cc:4841] TS 639c8bb1736b47da9fe60e4606c1f654 (127.5.213.194:43543): tablet 4812e968ce8a4d44af7d9fa2a3d4a027 (table test-table [id=e3bdebbcba764ff399b03596652ee1d6]) successfully deleted
I20260618 22:39:21.858942 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.858411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.0.0.1:52416)
0618 22:39:21.858677 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.858777 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260618 22:39:21.861358 13485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.838214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39000 (local address 127.5.213.193:43029)
0618 22:39:21.838415 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:21.838421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:21.845764 (+ 7343us) server_negotiation.cc:407] Connection header received
0618 22:39:21.845930 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:21.845936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:21.846005 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:21.846102 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:21.847934 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.849058 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.850750 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.851405 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.851491 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:21.854867 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:21.854881 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:21.854898 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:21.854932 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:21.857904 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:21.858738 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:21.858748 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:21.858751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:21.858901 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:21.859336 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:21.859341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:21.859344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:21.859853 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:21.860151 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:21.860758 (+ 607us) server_negotiation.cc:299] Negotiation successful
0618 22:39:21.861004 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 22:39:21.863242 13097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:21.864675 13512 ts_tablet_manager.cc:1918] T 4812e968ce8a4d44af7d9fa2a3d4a027 P b834563cb2c64ddeb15499cbd0c68a4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:39:21.864832 13512 log.cc:1196] T 4812e968ce8a4d44af7d9fa2a3d4a027 P b834563cb2c64ddeb15499cbd0c68a4c: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/4812e968ce8a4d44af7d9fa2a3d4a027
I20260618 22:39:21.865748 13512 ts_tablet_manager.cc:1939] T 4812e968ce8a4d44af7d9fa2a3d4a027 P b834563cb2c64ddeb15499cbd0c68a4c: Deleting consensus metadata
I20260618 22:39:21.868008 12717 catalog_manager.cc:4841] TS b834563cb2c64ddeb15499cbd0c68a4c (127.5.213.195:43255): tablet 4812e968ce8a4d44af7d9fa2a3d4a027 (table test-table [id=e3bdebbcba764ff399b03596652ee1d6]) successfully deleted
I20260618 22:39:21.875972 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.875544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33087 (local address 127.0.0.1:52426)
0618 22:39:21.875716 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:21.875813 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:33087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260618 22:39:21.957823 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 12906
I20260618 22:39:21.971565 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 13122
I20260618 22:39:21.986397 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:21.972233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39830 (local address 127.5.213.194:43543)
0618 22:39:21.972424 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:21.972430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:21.972517 (+ 87us) server_negotiation.cc:407] Connection header received
0618 22:39:21.972690 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:21.972695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:21.972763 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:21.972860 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:21.974042 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.974820 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.976161 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:21.976543 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:21.976635 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:21.979114 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:21.979128 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:21.979146 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:21.979175 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:21.982941 (+ 3766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:21.983844 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:21.983854 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:21.983858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:21.984014 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:21.984402 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:21.984408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:21.984412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:21.984839 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:21.985101 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:21.985853 (+ 752us) server_negotiation.cc:299] Negotiation successful
0618 22:39:21.986077 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:39:21.987038 13245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:22.103698 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 13122
I20260618 22:39:22.120675 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 13264
I20260618 22:39:22.133682 13518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:22.121379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51994 (local address 127.5.213.195:43255)
0618 22:39:22.121830 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:22.121835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:22.121849 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:39:22.121897 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:22.121900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:22.121955 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:22.122069 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:22.123382 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:22.124106 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:22.125337 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:22.125748 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:22.125865 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:22.128389 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:22.128403 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:22.128419 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:22.128454 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:22.131045 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:22.131703 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:22.131711 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:22.131714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:22.131836 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:22.132181 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:22.132185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:22.132187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:22.132554 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:22.132833 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:22.133139 (+ 306us) server_negotiation.cc:299] Negotiation successful
0618 22:39:22.133364 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":227,"threads_started":1}
I20260618 22:39:22.134287 13387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:22.226465 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 13264
I20260618 22:39:22.239859 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 12588
I20260618 22:39:22.240913 12761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:22.356508 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 12588
I20260618 22:39:22.372578 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 12684
I20260618 22:39:22.373590 12955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:22.483549 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 12684
2026-06-18T22:39:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9174 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:39:22 dist-test-slave-lnj3 krb5kdc[13524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:39:22 dist-test-slave-lnj3 krb5kdc[13524](info): set up 2 sockets
Jun 18 22:39:22 dist-test-slave-lnj3 krb5kdc[13524](info): commencing operation
krb5kdc: starting...
W20260618 22:39:23.491191 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:39:23 dist-test-slave-lnj3 krb5kdc[13524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:39:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:39:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.254.keytab.
Entry for principal oryx/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.254.keytab.
I20260618 22:39:23.689333 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:36989
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.254.keytab
--principal=oryx/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:36989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:24.008339 13537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:24.008626 13537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:24.008762 13537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:24.016069 13537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:39:24.016180 13537 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260618 22:39:24.016250 13537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:24.016355 13537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:24.016433 13537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:39:24.016534 13537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:39:24.026286 13537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:36989
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.254.keytab
--principal=oryx/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:36989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:24.027576 13537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:24.028901 13537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:24.040192 13542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:24.040230 13545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:24.040153 13543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:24.041038 13537 server_base.cc:1034] running on GCE node
I20260618 22:39:24.041887 13537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:24.043287 13537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:24.044509 13537 hybrid_clock.cc:648] HybridClock initialized: now 1781822364044445 us; error 38 us; skew 500 ppm
I20260618 22:39:24.044868 13537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:24 dist-test-slave-lnj3 krb5kdc[13524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822364, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:24.052623 13537 init.cc:377] Logged in from keytab as oryx/127.5.213.254@KRBTEST.COM (short username oryx)
I20260618 22:39:24.054116 13537 webserver.cc:460] Webserver started at http://127.5.213.254:41307/ using document root <none> and password file <none>
I20260618 22:39:24.054868 13537 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:24.054975 13537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:24.055298 13537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:24.058393 13537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "3284b18e08334919b3a7fd0c05a13aca"
format_stamp: "Formatted at 2026-06-18 22:39:24 on dist-test-slave-lnj3"
server_key: "da3082a91d3733cf84503ebd39cba39d"
server_key_iv: "fd71477b424566249d1b8d022811534e"
server_key_version: "encryptionkey@0"
I20260618 22:39:24.059234 13537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "3284b18e08334919b3a7fd0c05a13aca"
format_stamp: "Formatted at 2026-06-18 22:39:24 on dist-test-slave-lnj3"
server_key: "da3082a91d3733cf84503ebd39cba39d"
server_key_iv: "fd71477b424566249d1b8d022811534e"
server_key_version: "encryptionkey@0"
I20260618 22:39:24.064196 13537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260618 22:39:24.067562 13552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:24.068795 13537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260618 22:39:24.068948 13537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "3284b18e08334919b3a7fd0c05a13aca"
format_stamp: "Formatted at 2026-06-18 22:39:24 on dist-test-slave-lnj3"
server_key: "da3082a91d3733cf84503ebd39cba39d"
server_key_iv: "fd71477b424566249d1b8d022811534e"
server_key_version: "encryptionkey@0"
I20260618 22:39:24.069144 13537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:24.100126 13537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:24.104215 13537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:24.104466 13537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:24.123587 13537 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:36989
I20260618 22:39:24.123603 13613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:36989 every 8 connection(s)
I20260618 22:39:24.124920 13537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:39:24.127045 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 13537
I20260618 22:39:24.127245 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:39:24.128002 5975 external_mini_cluster.cc:1428] Setting key f01aa883371d19e5ae7a149713e189b7
I20260618 22:39:24.129237 13614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 18 22:39:24 dist-test-slave-lnj3 krb5kdc[13524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM
I20260618 22:39:24.142671 13614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca: Bootstrap starting.
I20260618 22:39:24.145715 13614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:24.146955 13614 log.cc:824] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:24.149956 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:24.130009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.5.213.254:36989)
0618 22:39:24.130736 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:24.130747 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:24.130787 (+ 40us) server_negotiation.cc:407] Connection header received
0618 22:39:24.132298 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:24.132319 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:24.132705 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:24.133025 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:24.134659 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:24.135982 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:24.137635 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:24.138303 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:24.138427 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:24.143033 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:24.143049 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:24.143063 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:24.143097 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:24.145891 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:24.146620 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:24.146650 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:24.146659 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:24.146857 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:24.147264 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:24.147268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:24.147270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:24.148011 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:24.148318 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:24.148608 (+ 290us) server_negotiation.cc:299] Negotiation successful
0618 22:39:24.148952 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":323,"threads_started":1}
I20260618 22:39:24.150051 13614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca: No bootstrap required, opened a new log
I20260618 22:39:24.157832 13614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3284b18e08334919b3a7fd0c05a13aca" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36989 } }
I20260618 22:39:24.158190 13614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:24.158294 13614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3284b18e08334919b3a7fd0c05a13aca, State: Initialized, Role: FOLLOWER
I20260618 22:39:24.158885 13614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3284b18e08334919b3a7fd0c05a13aca" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36989 } }
I20260618 22:39:24.159109 13614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:39:24.159242 13614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:39:24.159403 13614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:24.161566 13614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3284b18e08334919b3a7fd0c05a13aca" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36989 } }
I20260618 22:39:24.162042 13614 leader_election.cc:302] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3284b18e08334919b3a7fd0c05a13aca; no voters:
I20260618 22:39:24.162608 13614 leader_election.cc:288] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:39:24.162724 13619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:24.163910 13619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca [term 1 LEADER]: Becoming Leader. State: Replica: 3284b18e08334919b3a7fd0c05a13aca, State: Running, Role: LEADER
I20260618 22:39:24.164391 13619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3284b18e08334919b3a7fd0c05a13aca" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36989 } }
I20260618 22:39:24.164716 13614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:39:24.166884 13620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3284b18e08334919b3a7fd0c05a13aca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3284b18e08334919b3a7fd0c05a13aca" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36989 } } }
I20260618 22:39:24.167246 13620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:24.167985 13621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3284b18e08334919b3a7fd0c05a13aca. Latest consensus state: current_term: 1 leader_uuid: "3284b18e08334919b3a7fd0c05a13aca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3284b18e08334919b3a7fd0c05a13aca" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36989 } } }
I20260618 22:39:24.168375 13621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:24.173811 13625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:39:24.179342 13625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:39:24.187808 13625 catalog_manager.cc:1347] Generated new cluster ID: 4011061169ee4b43838e4ce94036266e
I20260618 22:39:24.187923 13625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:39:24.200930 13625 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:39:24.202222 13625 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:39:24.215466 13625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca: Generated new TSK 0
I20260618 22:39:24.216320 13625 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.193.keytab.
Entry for principal oryx/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.193.keytab.
I20260618 22:39:24.281353 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36989
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.193.keytab
--principal=oryx/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:24.608732 13640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:24.609030 13640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:24.609155 13640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:24.616325 13640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:24.616465 13640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:24.616575 13640 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:39:24.616688 13640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:39:24.627483 13640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.193.keytab
--principal=oryx/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.213.254:36989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:24.628746 13640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:24.630008 13640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:24.642275 13645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:24.642232 13648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:24.642295 13646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:24.643306 13640 server_base.cc:1034] running on GCE node
I20260618 22:39:24.643875 13640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:24.644714 13640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:24.645920 13640 hybrid_clock.cc:648] HybridClock initialized: now 1781822364645859 us; error 40 us; skew 500 ppm
I20260618 22:39:24.646186 13640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:24 dist-test-slave-lnj3 krb5kdc[13524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822364, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:24.649284 13640 init.cc:377] Logged in from keytab as oryx/127.5.213.193@KRBTEST.COM (short username oryx)
I20260618 22:39:24.650697 13640 webserver.cc:460] Webserver started at http://127.5.213.193:43049/ using document root <none> and password file <none>
I20260618 22:39:24.651417 13640 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:24.651503 13640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:24.651800 13640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:24.654539 13640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "1bdb809d25fe458b98b389dfd4cb2c3b"
format_stamp: "Formatted at 2026-06-18 22:39:24 on dist-test-slave-lnj3"
server_key: "8d243d3eca020ca48f71eab91521b6ca"
server_key_iv: "d6563631cc763aa5f7572d7d03b95416"
server_key_version: "encryptionkey@0"
I20260618 22:39:24.655326 13640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "1bdb809d25fe458b98b389dfd4cb2c3b"
format_stamp: "Formatted at 2026-06-18 22:39:24 on dist-test-slave-lnj3"
server_key: "8d243d3eca020ca48f71eab91521b6ca"
server_key_iv: "d6563631cc763aa5f7572d7d03b95416"
server_key_version: "encryptionkey@0"
I20260618 22:39:24.660274 13640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260618 22:39:24.663612 13655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:24.664880 13640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260618 22:39:24.665024 13640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "1bdb809d25fe458b98b389dfd4cb2c3b"
format_stamp: "Formatted at 2026-06-18 22:39:24 on dist-test-slave-lnj3"
server_key: "8d243d3eca020ca48f71eab91521b6ca"
server_key_iv: "d6563631cc763aa5f7572d7d03b95416"
server_key_version: "encryptionkey@0"
I20260618 22:39:24.665257 13640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:24.681001 13640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:24.684942 13640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:24.685178 13640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:24.687651 13640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:24.687729 13640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:24.687860 13640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:24.687953 13640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 18 22:39:24 dist-test-slave-lnj3 krb5kdc[13524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822364, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM
I20260618 22:39:24.710239 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:24.688049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:47533 (local address 127.5.213.254:36989)
0618 22:39:24.688758 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:24.688764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:24.689683 (+ 919us) server_negotiation.cc:407] Connection header received
0618 22:39:24.692026 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:24.692033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:24.692117 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:24.692240 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:24.694162 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:24.695133 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:24.698019 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:24.698451 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:24.698548 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:24.703479 (+ 4931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:24.703484 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:24.703487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:24.703520 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:24.706660 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:24.707557 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:24.707562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:24.707566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:24.707726 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:24.708258 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:24.708264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:24.708267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:24.708655 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:24.708802 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:24.709657 (+ 855us) server_negotiation.cc:299] Negotiation successful
0618 22:39:24.709898 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":484,"threads_started":1}
I20260618 22:39:24.711460 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:24.688684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:36989 (local address 127.5.213.193:47533)
0618 22:39:24.689514 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:24.689560 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:24.691624 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:24.692494 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:24.692503 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:24.693165 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:24.693986 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:24.694013 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:24.695346 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:24.695352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:24.697859 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:24.697876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:24.698710 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:24.698718 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:24.699054 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:24.699950 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:24.699986 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:24.703227 (+ 3241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:24.706813 (+ 3586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:24.706826 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:24.706839 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:24.707441 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:24.707879 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:24.707885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:24.707889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:24.708142 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:24.708795 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:24.708803 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:24.709196 (+ 393us) client_negotiation.cc:769] Sending connection context
0618 22:39:24.709553 (+ 357us) client_negotiation.cc:240] Negotiation successful
0618 22:39:24.710111 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":349,"threads_started":1}
I20260618 22:39:24.734187 13640 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:40341
I20260618 22:39:24.734272 13775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:40341 every 8 connection(s)
I20260618 22:39:24.734915 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:24.721411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:46965 (local address 127.5.213.254:36989)
0618 22:39:24.721595 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:24.721602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:24.722123 (+ 521us) server_negotiation.cc:407] Connection header received
0618 22:39:24.722267 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:24.722273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:24.722356 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:24.722460 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:24.723465 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:24.724593 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:24.726489 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:24.727157 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:24.727285 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:24.728927 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:24.728933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:24.728939 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:24.728974 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:24.732152 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:24.732818 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:24.732823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:24.732826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:24.732966 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:24.733376 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:24.733382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:24.733386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:24.733727 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:24.733875 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:24.734354 (+ 479us) server_negotiation.cc:299] Negotiation successful
0618 22:39:24.734557 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:39:24.734925 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:24.721424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:36989 (local address 127.5.213.193:46965)
0618 22:39:24.722033 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:24.722051 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:24.722166 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:24.722502 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:24.722505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:24.723024 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:24.723322 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:24.723331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:24.724777 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:24.724783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:24.726349 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:24.726370 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:24.727321 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:24.727326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:24.727432 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:24.728044 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:24.728064 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:24.728769 (+ 705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:24.732306 (+ 3537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:24.732310 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:24.732313 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:24.732715 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:24.733071 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:24.733076 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:24.733097 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:24.733264 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:24.733853 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:24.733858 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:24.734065 (+ 207us) client_negotiation.cc:769] Sending connection context
0618 22:39:24.734247 (+ 182us) client_negotiation.cc:240] Negotiation successful
0618 22:39:24.734555 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":333,"threads_started":1}
I20260618 22:39:24.735975 13640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:39:24.738420 13776 heartbeater.cc:347] Connected to a master server at 127.5.213.254:36989
I20260618 22:39:24.738688 13776 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:24.739305 13776 heartbeater.cc:511] Master 127.5.213.254:36989 requested a full tablet report, sending...
I20260618 22:39:24.741021 13569 ts_manager.cc:194] Registered new tserver with Master: 1bdb809d25fe458b98b389dfd4cb2c3b (127.5.213.193:40341)
I20260618 22:39:24.742380 13569 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:47533
I20260618 22:39:24.743113 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 13640
I20260618 22:39:24.743290 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:39:24.744025 5975 external_mini_cluster.cc:1428] Setting key a70e1714e028268ea55bc0933f0b9ce0
WARNING: no policy specified for oryx/127.5.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.194.keytab.
Entry for principal oryx/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.194.keytab.
I20260618 22:39:24.785494 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36989
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.194.keytab
--principal=oryx/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:25.096526 13780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:25.096829 13780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:25.096964 13780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:25.104146 13780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:25.104260 13780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:25.104367 13780 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:39:25.104467 13780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:39:25.114140 13780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.194.keytab
--principal=oryx/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.213.254:36989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:25.115463 13780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:25.116830 13780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:25.130716 13785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:25.130733 13786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:25.131246 13788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:25.131850 13780 server_base.cc:1034] running on GCE node
I20260618 22:39:25.132480 13780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:25.133416 13780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:25.134663 13780 hybrid_clock.cc:648] HybridClock initialized: now 1781822365134550 us; error 59 us; skew 500 ppm
I20260618 22:39:25.134974 13780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:25 dist-test-slave-lnj3 krb5kdc[13524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822365, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:25.138100 13780 init.cc:377] Logged in from keytab as oryx/127.5.213.194@KRBTEST.COM (short username oryx)
I20260618 22:39:25.139557 13780 webserver.cc:460] Webserver started at http://127.5.213.194:39237/ using document root <none> and password file <none>
I20260618 22:39:25.140225 13780 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:25.140323 13780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:25.140612 13780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:25.143462 13780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "2c749157c20c458bb3c63cd5b01247b9"
format_stamp: "Formatted at 2026-06-18 22:39:25 on dist-test-slave-lnj3"
server_key: "5d4ca0f62e0d9e231d9c99cb48d4fb48"
server_key_iv: "649307287327ed66886c2ef1c8029097"
server_key_version: "encryptionkey@0"
I20260618 22:39:25.144285 13780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "2c749157c20c458bb3c63cd5b01247b9"
format_stamp: "Formatted at 2026-06-18 22:39:25 on dist-test-slave-lnj3"
server_key: "5d4ca0f62e0d9e231d9c99cb48d4fb48"
server_key_iv: "649307287327ed66886c2ef1c8029097"
server_key_version: "encryptionkey@0"
I20260618 22:39:25.149238 13780 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:39:25.152546 13795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:25.153820 13780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:39:25.153964 13780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "2c749157c20c458bb3c63cd5b01247b9"
format_stamp: "Formatted at 2026-06-18 22:39:25 on dist-test-slave-lnj3"
server_key: "5d4ca0f62e0d9e231d9c99cb48d4fb48"
server_key_iv: "649307287327ed66886c2ef1c8029097"
server_key_version: "encryptionkey@0"
I20260618 22:39:25.154151 13780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:25.174401 13780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:25.178427 13780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:25.178719 13780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:25.181280 13780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:25.181365 13780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:25.181494 13780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:25.181550 13780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:39:25 dist-test-slave-lnj3 krb5kdc[13524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822365, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM
I20260618 22:39:25.203433 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:25.181808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:60305 (local address 127.5.213.254:36989)
0618 22:39:25.181989 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:25.181995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:25.183480 (+ 1485us) server_negotiation.cc:407] Connection header received
0618 22:39:25.185558 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:25.185565 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:25.185666 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:25.185744 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:25.188390 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:25.189608 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:25.191544 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:25.192084 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:25.192198 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:25.196448 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:25.196456 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:25.196462 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:25.196512 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:25.199896 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:25.200855 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:25.200860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:25.200864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:25.200989 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:25.201490 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:25.201495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:25.201499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:25.201897 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:25.202060 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:25.202861 (+ 801us) server_negotiation.cc:299] Negotiation successful
0618 22:39:25.203098 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260618 22:39:25.204516 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:25.182411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:36989 (local address 127.5.213.194:60305)
0618 22:39:25.183286 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:25.183361 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:25.185189 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:25.186428 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:25.186441 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:25.187270 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:25.188130 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:25.188160 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:25.189778 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:25.189782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:25.191404 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:25.191416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:25.192317 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:25.192324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:25.192586 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:25.193374 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:25.193415 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:25.196160 (+ 2745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:25.200050 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:25.200060 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:25.200082 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:25.200728 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:25.201105 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:25.201114 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:25.201116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:25.201344 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:25.202074 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:25.202083 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:25.202469 (+ 386us) client_negotiation.cc:769] Sending connection context
0618 22:39:25.202758 (+ 289us) client_negotiation.cc:240] Negotiation successful
0618 22:39:25.203317 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":379,"threads_started":1}
I20260618 22:39:25.223556 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:25.209897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:58157 (local address 127.5.213.254:36989)
0618 22:39:25.210063 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:25.210070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:25.210739 (+ 669us) server_negotiation.cc:407] Connection header received
0618 22:39:25.210928 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:25.210934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:25.210998 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:25.211073 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:25.212274 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:25.213217 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:25.215313 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:25.215870 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:25.215959 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:25.218136 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:25.218141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:25.218144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:25.218172 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:25.220506 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:25.221329 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:25.221334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:25.221336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:25.221469 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:25.221845 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:25.221849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:25.221851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:25.222256 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:25.222390 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:25.223012 (+ 622us) server_negotiation.cc:299] Negotiation successful
0618 22:39:25.223214 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:39:25.223693 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:25.209950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:36989 (local address 127.5.213.194:58157)
0618 22:39:25.210614 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:25.210660 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:25.210800 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:25.211130 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:25.211135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:25.211733 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:25.212147 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:25.212158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:25.213375 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:25.213382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:25.215166 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:25.215181 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:25.216037 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:25.216044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:25.216237 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:25.216896 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:25.216919 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:25.217947 (+ 1028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:25.220630 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:25.220636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:25.220640 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:25.221196 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:25.221574 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:25.221579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:25.221583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:25.221741 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:25.222373 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:25.222378 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:25.222606 (+ 228us) client_negotiation.cc:769] Sending connection context
0618 22:39:25.222909 (+ 303us) client_negotiation.cc:240] Negotiation successful
0618 22:39:25.223293 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":354,"threads_started":1}
I20260618 22:39:25.234887 13780 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:35219
I20260618 22:39:25.234926 13914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:35219 every 8 connection(s)
I20260618 22:39:25.236050 13780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:39:25.238013 13915 heartbeater.cc:347] Connected to a master server at 127.5.213.254:36989
I20260618 22:39:25.238209 13915 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:25.238801 13915 heartbeater.cc:511] Master 127.5.213.254:36989 requested a full tablet report, sending...
I20260618 22:39:25.240001 13569 ts_manager.cc:194] Registered new tserver with Master: 2c749157c20c458bb3c63cd5b01247b9 (127.5.213.194:35219)
I20260618 22:39:25.240782 13569 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:60305
I20260618 22:39:25.244654 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 13780
I20260618 22:39:25.244801 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:39:25.245149 5975 external_mini_cluster.cc:1428] Setting key 77668adc0427b40937b6b3e162fed162
WARNING: no policy specified for oryx/127.5.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.195.keytab.
Entry for principal oryx/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.195.keytab.
I20260618 22:39:25.285633 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36989
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.195.keytab
--principal=oryx/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:25.594658 13919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:25.594969 13919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:25.595099 13919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:25.602216 13919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:25.602337 13919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:25.602490 13919 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:39:25.602607 13919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:39:25.612463 13919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.195.keytab
--principal=oryx/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.213.254:36989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:25.613715 13919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:25.615070 13919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:25.628414 13924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:25.628510 13927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:25.628510 13925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:25.629438 13919 server_base.cc:1034] running on GCE node
I20260618 22:39:25.630164 13919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:25.631210 13919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:25.632460 13919 hybrid_clock.cc:648] HybridClock initialized: now 1781822365632380 us; error 47 us; skew 500 ppm
I20260618 22:39:25.632784 13919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:25 dist-test-slave-lnj3 krb5kdc[13524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822365, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:25.636026 13919 init.cc:377] Logged in from keytab as oryx/127.5.213.195@KRBTEST.COM (short username oryx)
I20260618 22:39:25.637485 13919 webserver.cc:460] Webserver started at http://127.5.213.195:33625/ using document root <none> and password file <none>
I20260618 22:39:25.638152 13919 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:25.638234 13919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:25.638527 13919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:25.641381 13919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23"
format_stamp: "Formatted at 2026-06-18 22:39:25 on dist-test-slave-lnj3"
server_key: "2d68ef98fa40c117dd2def1916aae26a"
server_key_iv: "41ba8506655a185011d4465cbbe3d6c0"
server_key_version: "encryptionkey@0"
I20260618 22:39:25.642166 13919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23"
format_stamp: "Formatted at 2026-06-18 22:39:25 on dist-test-slave-lnj3"
server_key: "2d68ef98fa40c117dd2def1916aae26a"
server_key_iv: "41ba8506655a185011d4465cbbe3d6c0"
server_key_version: "encryptionkey@0"
I20260618 22:39:25.647274 13919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260618 22:39:25.650688 13934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:25.652045 13919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:39:25.652190 13919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23"
format_stamp: "Formatted at 2026-06-18 22:39:25 on dist-test-slave-lnj3"
server_key: "2d68ef98fa40c117dd2def1916aae26a"
server_key_iv: "41ba8506655a185011d4465cbbe3d6c0"
server_key_version: "encryptionkey@0"
I20260618 22:39:25.652385 13919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:25.666535 13919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:25.670667 13919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:25.670913 13919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:25.673506 13919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:25.673588 13919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:25.673722 13919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:25.673799 13919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:39:25 dist-test-slave-lnj3 krb5kdc[13524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822365, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM
I20260618 22:39:25.697898 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:25.674828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:36989 (local address 127.5.213.195:41663)
0618 22:39:25.675672 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:25.675723 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:25.677856 (+ 2133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:25.678867 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:25.678874 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:25.679622 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:25.680391 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:25.680423 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:25.681944 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:25.681953 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:25.683812 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:25.683841 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:25.684740 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:25.684748 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:25.685025 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:25.685810 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:25.685843 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:25.689335 (+ 3492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:25.693026 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:25.693046 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:25.693064 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:25.693632 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:25.694064 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:25.694073 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:25.694076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:25.694339 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:25.694981 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:25.694990 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:25.695467 (+ 477us) client_negotiation.cc:769] Sending connection context
0618 22:39:25.695811 (+ 344us) client_negotiation.cc:240] Negotiation successful
0618 22:39:25.696357 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":349,"threads_started":1}
I20260618 22:39:25.698964 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:25.674221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:41663 (local address 127.5.213.254:36989)
0618 22:39:25.674402 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:25.674408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:25.675852 (+ 1444us) server_negotiation.cc:407] Connection header received
0618 22:39:25.678215 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:25.678219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:25.678284 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:25.678337 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:25.680606 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:25.681777 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:25.683965 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:25.684517 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:25.684635 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:25.689561 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:25.689566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:25.689569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:25.689602 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:25.692883 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:25.693765 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:25.693770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:25.693774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:25.693933 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:25.694453 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:25.694458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:25.694461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:25.694843 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:25.694977 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:25.698450 (+ 3473us) server_negotiation.cc:299] Negotiation successful
0618 22:39:25.698724 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260618 22:39:25.707185 13919 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:37043
I20260618 22:39:25.707226 14053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:37043 every 8 connection(s)
I20260618 22:39:25.708555 13919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:39:25.710457 14054 heartbeater.cc:347] Connected to a master server at 127.5.213.254:36989
I20260618 22:39:25.710676 14054 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:25.711347 14054 heartbeater.cc:511] Master 127.5.213.254:36989 requested a full tablet report, sending...
I20260618 22:39:25.712620 13569 ts_manager.cc:194] Registered new tserver with Master: 1d7cdbc2b91b4a5ba6f70c4605d8cc23 (127.5.213.195:37043)
I20260618 22:39:25.713613 13569 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:41663
I20260618 22:39:25.714896 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 13919
I20260618 22:39:25.715063 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:39:25.715570 5975 external_mini_cluster.cc:1428] Setting key 0742c5b2d06aeb3df707c5333c80c840
I20260618 22:39:25.718312 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:39:25.718492 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:25.705112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:36569 (local address 127.5.213.254:36989)
0618 22:39:25.705288 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:25.705292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:25.706294 (+ 1002us) server_negotiation.cc:407] Connection header received
0618 22:39:25.706520 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:25.706526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:25.706613 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:25.706729 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:25.707918 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:25.708605 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:25.710182 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:25.710759 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:25.710869 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:25.712648 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:25.712653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:25.712656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:25.712686 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:25.715827 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:25.716618 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:25.716623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:25.716626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:25.716783 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:25.717180 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:25.717186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:25.717190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:25.717458 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:25.717591 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:25.717960 (+ 369us) server_negotiation.cc:299] Negotiation successful
0618 22:39:25.718151 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:39:25.718547 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:25.705364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:36989 (local address 127.5.213.195:36569)
0618 22:39:25.706157 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:25.706184 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:25.706395 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:25.706770 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:25.706775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:25.707435 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:25.707767 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:25.707781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:25.708729 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:25.708734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:25.710048 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:25.710061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:25.710927 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:25.710933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:25.711118 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:25.711719 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:25.711735 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:25.712480 (+ 745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:25.715973 (+ 3493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:25.715978 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:25.715982 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:25.716371 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:25.716896 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:25.716900 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:25.716904 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:25.717063 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:25.717576 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:25.717581 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:25.717737 (+ 156us) client_negotiation.cc:769] Sending connection context
0618 22:39:25.717854 (+ 117us) client_negotiation.cc:240] Negotiation successful
0618 22:39:25.718139 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":2,"thread_start_us":377,"threads_started":1}
I20260618 22:39:25.745340 13776 heartbeater.cc:503] Master 127.5.213.254:36989 was elected leader, sending a full tablet report...
Jun 18 22:39:25 dist-test-slave-lnj3 krb5kdc[13524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822365, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 22:39:25 dist-test-slave-lnj3 krb5kdc[13524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:25 dist-test-slave-lnj3 krb5kdc[13524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:26.243279 13915 heartbeater.cc:503] Master 127.5.213.254:36989 was elected leader, sending a full tablet report...
I20260618 22:39:26.716226 14054 heartbeater.cc:503] Master 127.5.213.254:36989 was elected leader, sending a full tablet report...
W20260618 22:39:26.735391 14062 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.976s user 0.006s sys 0.007s
W20260618 22:39:26.736505 13664 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:25.754113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46548 (local address 127.5.213.254:36989)
0618 22:39:25.754271 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:25.754277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:25.754586 (+ 309us) server_negotiation.cc:407] Connection header received
0618 22:39:25.754787 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:25.754791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:25.754852 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:25.754931 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:25.756249 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:25.757085 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:25.758356 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:25.758834 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:25.758926 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:26.736208 (+977282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46548: BlockingRecv error: recv got EOF from 127.0.0.1:46548 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 18 22:39:26 dist-test-slave-lnj3 krb5kdc[13524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822365, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM
I20260618 22:39:26.753237 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:26.740203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46560 (local address 127.5.213.254:36989)
0618 22:39:26.740398 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:26.740404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:26.740666 (+ 262us) server_negotiation.cc:407] Connection header received
0618 22:39:26.740771 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:26.740774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:26.740874 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:26.740960 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:26.742113 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.742944 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.744184 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.744606 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.744700 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:26.748309 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:26.748316 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:26.748320 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:26.748353 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:26.750956 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:26.751636 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:26.751641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:26.751643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:26.751754 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:26.752082 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:26.752087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:26.752089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:26.752380 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:26.752620 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:26.752842 (+ 222us) server_negotiation.cc:299] Negotiation successful
0618 22:39:26.753006 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 22:39:26.757248 13569 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:39:26.763859 13569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:39:26.796712 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:26.775805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43510 (local address 127.5.213.193:40341)
0618 22:39:26.787692 (+ 11887us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:26.787702 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:26.788628 (+ 926us) server_negotiation.cc:407] Connection header received
0618 22:39:26.788740 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:26.788747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:26.789019 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:26.790032 (+ 1013us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:26.790249 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.791865 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.794458 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.795656 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.795770 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:26.795971 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:26.796251 (+ 280us) server_negotiation.cc:299] Negotiation successful
0618 22:39:26.796421 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11703,"thread_start_us":11544,"threads_started":1}
I20260618 22:39:26.796768 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:26.775616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:40341 (local address 127.0.0.1:43510)
0618 22:39:26.788151 (+ 12535us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:26.788200 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:26.788413 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:26.789297 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:26.789306 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:26.789343 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:26.789818 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:26.789830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.792022 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.792028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:26.794323 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:26.794332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.795851 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.795857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:26.796024 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:26.796072 (+ 48us) client_negotiation.cc:769] Sending connection context
0618 22:39:26.796192 (+ 120us) client_negotiation.cc:240] Negotiation successful
0618 22:39:26.796372 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12273,"spinlock_wait_cycles":3968,"thread_start_us":12118,"threads_started":1}
I20260618 22:39:26.798269 14075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:26.775914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:37043 (local address 127.0.0.1:43116)
0618 22:39:26.789195 (+ 13281us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:26.789212 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:26.789417 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:26.790174 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:26.790178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:26.790200 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:26.790584 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:26.790597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.793771 (+ 3174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.793775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:26.795632 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:26.795645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.797586 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.797590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:26.797744 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:26.797766 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:39:26.797818 (+ 52us) client_negotiation.cc:240] Negotiation successful
0618 22:39:26.797902 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13141,"mutex_wait_us":19,"spinlock_wait_cycles":2816,"thread_start_us":13010,"threads_started":1}
I20260618 22:39:26.798826 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:26.776349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43116 (local address 127.5.213.195:37043)
0618 22:39:26.788553 (+ 12204us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:26.788564 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:26.789306 (+ 742us) server_negotiation.cc:407] Connection header received
0618 22:39:26.789600 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:26.789608 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:26.789842 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:26.789998 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:26.790819 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.792870 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.795782 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.797053 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.798094 (+ 1041us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:26.798300 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:26.798380 (+ 80us) server_negotiation.cc:299] Negotiation successful
0618 22:39:26.798533 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12037,"thread_start_us":12221,"threads_started":1}
I20260618 22:39:26.800819 13706 tablet_service.cc:1467] Processing CreateTablet for tablet 826419e1a9c74db3af5025a13f9af266 (DEFAULT_TABLE table=test-table [id=9a3e32727a0d4e198d4ec552d1ec5fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:26.801931 14078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:26.790270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:35219 (local address 127.0.0.1:54286)
0618 22:39:26.793829 (+ 3559us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:26.793846 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:26.793961 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:26.794555 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:26.794559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:26.794583 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:26.794994 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:26.795005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.797484 (+ 2479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.797490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:26.799626 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:26.799641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.801139 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.801146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:26.801341 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:26.801364 (+ 23us) client_negotiation.cc:769] Sending connection context
0618 22:39:26.801467 (+ 103us) client_negotiation.cc:240] Negotiation successful
0618 22:39:26.801591 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3432,"mutex_wait_us":9,"thread_start_us":3284,"threads_started":1}
I20260618 22:39:26.801929 14077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:26.788886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54286 (local address 127.5.213.194:35219)
0618 22:39:26.793517 (+ 4631us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:26.793525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:26.793971 (+ 446us) server_negotiation.cc:407] Connection header received
0618 22:39:26.794139 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:26.794147 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:26.794370 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:26.794560 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:26.795184 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.797369 (+ 2185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.799812 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.800922 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.801048 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:26.801236 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:26.801507 (+ 271us) server_negotiation.cc:299] Negotiation successful
0618 22:39:26.801676 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4484,"thread_start_us":4359,"threads_started":1}
I20260618 22:39:26.802718 13706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 826419e1a9c74db3af5025a13f9af266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:26.802155 13984 tablet_service.cc:1467] Processing CreateTablet for tablet 826419e1a9c74db3af5025a13f9af266 (DEFAULT_TABLE table=test-table [id=9a3e32727a0d4e198d4ec552d1ec5fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:26.803560 13984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 826419e1a9c74db3af5025a13f9af266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:26.804709 13845 tablet_service.cc:1467] Processing CreateTablet for tablet 826419e1a9c74db3af5025a13f9af266 (DEFAULT_TABLE table=test-table [id=9a3e32727a0d4e198d4ec552d1ec5fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:26.805680 13845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 826419e1a9c74db3af5025a13f9af266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:26.816136 14079 tablet_bootstrap.cc:492] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9: Bootstrap starting.
I20260618 22:39:26.819255 14079 tablet_bootstrap.cc:654] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:26.819269 14081 tablet_bootstrap.cc:492] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b: Bootstrap starting.
I20260618 22:39:26.820662 14079 log.cc:824] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:26.822701 14081 tablet_bootstrap.cc:654] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:26.823323 14080 tablet_bootstrap.cc:492] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Bootstrap starting.
I20260618 22:39:26.824808 14081 log.cc:824] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:26.825439 14079 tablet_bootstrap.cc:492] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9: No bootstrap required, opened a new log
I20260618 22:39:26.825699 14079 ts_tablet_manager.cc:1397] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260618 22:39:26.826560 14080 tablet_bootstrap.cc:654] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:26.827185 14081 tablet_bootstrap.cc:492] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b: No bootstrap required, opened a new log
I20260618 22:39:26.827399 14081 ts_tablet_manager.cc:1397] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260618 22:39:26.827716 14080 log.cc:824] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:26.830404 14080 tablet_bootstrap.cc:492] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: No bootstrap required, opened a new log
I20260618 22:39:26.830679 14080 ts_tablet_manager.cc:1397] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260618 22:39:26.834524 14081 raft_consensus.cc:348] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } }
I20260618 22:39:26.834911 14081 raft_consensus.cc:374] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:26.835031 14081 raft_consensus.cc:729] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bdb809d25fe458b98b389dfd4cb2c3b, State: Initialized, Role: FOLLOWER
I20260618 22:39:26.835278 14079 raft_consensus.cc:348] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } }
I20260618 22:39:26.835678 14079 raft_consensus.cc:374] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:26.835592 14081 consensus_queue.cc:260] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } }
I20260618 22:39:26.835764 14079 raft_consensus.cc:729] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c749157c20c458bb3c63cd5b01247b9, State: Initialized, Role: FOLLOWER
I20260618 22:39:26.836340 14079 consensus_queue.cc:260] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } }
I20260618 22:39:26.838145 14079 ts_tablet_manager.cc:1428] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9: Time spent starting tablet: real 0.012s user 0.012s sys 0.002s
I20260618 22:39:26.839637 14081 ts_tablet_manager.cc:1428] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260618 22:39:26.840963 14080 raft_consensus.cc:348] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } }
I20260618 22:39:26.841391 14080 raft_consensus.cc:374] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:26.841490 14080 raft_consensus.cc:729] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d7cdbc2b91b4a5ba6f70c4605d8cc23, State: Initialized, Role: FOLLOWER
I20260618 22:39:26.842201 14080 consensus_queue.cc:260] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } }
I20260618 22:39:26.844707 14080 ts_tablet_manager.cc:1428] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Time spent starting tablet: real 0.014s user 0.009s sys 0.002s
I20260618 22:39:26.864907 14086 raft_consensus.cc:482] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:39:26.865271 14086 raft_consensus.cc:504] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } }
I20260618 22:39:26.866994 14086 leader_election.cc:288] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bdb809d25fe458b98b389dfd4cb2c3b (127.5.213.193:40341), 1d7cdbc2b91b4a5ba6f70c4605d8cc23 (127.5.213.195:37043)
I20260618 22:39:26.874589 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:26.867449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:53751 (local address 127.5.213.195:37043)
0618 22:39:26.867641 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:26.867646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:26.868283 (+ 637us) server_negotiation.cc:407] Connection header received
0618 22:39:26.868462 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:26.868467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:26.868541 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:26.868618 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:26.869206 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.870015 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.872464 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.873642 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.873813 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:26.873887 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:26.874263 (+ 376us) server_negotiation.cc:299] Negotiation successful
0618 22:39:26.874359 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260618 22:39:26.874788 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:26.867428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:37043 (local address 127.5.213.194:53751)
0618 22:39:26.868186 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:26.868204 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:26.868329 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:26.868644 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:26.868648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:26.868671 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:26.869055 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:26.869068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.870127 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.870132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:26.872317 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:26.872331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.873877 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.873884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:26.874080 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:26.874120 (+ 40us) client_negotiation.cc:769] Sending connection context
0618 22:39:26.874216 (+ 96us) client_negotiation.cc:240] Negotiation successful
0618 22:39:26.874351 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":5,"thread_start_us":441,"threads_started":1}
I20260618 22:39:26.874827 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:26.867312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:41817 (local address 127.5.213.193:40341)
0618 22:39:26.867521 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:26.867528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:26.868188 (+ 660us) server_negotiation.cc:407] Connection header received
0618 22:39:26.868359 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:26.868363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:26.868433 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:26.868531 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:26.869264 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.870123 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.872769 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.873950 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.874096 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:26.874144 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:26.874464 (+ 320us) server_negotiation.cc:299] Negotiation successful
0618 22:39:26.874582 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260618 22:39:26.875056 14088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:26.867222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:40341 (local address 127.5.213.194:41817)
0618 22:39:26.868062 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:26.868084 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:26.868228 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:26.868604 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:26.868608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:26.868638 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:26.869131 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:26.869142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.870669 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.870674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:26.872618 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:26.872631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.874148 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.874153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:26.874318 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:26.874346 (+ 28us) client_negotiation.cc:769] Sending connection context
0618 22:39:26.874423 (+ 77us) client_negotiation.cc:240] Negotiation successful
0618 22:39:26.874592 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":24,"thread_start_us":518,"threads_started":1}
I20260618 22:39:26.875854 14004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "826419e1a9c74db3af5025a13f9af266" candidate_uuid: "2c749157c20c458bb3c63cd5b01247b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" is_pre_election: true
I20260618 22:39:26.876173 13726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "826419e1a9c74db3af5025a13f9af266" candidate_uuid: "2c749157c20c458bb3c63cd5b01247b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" is_pre_election: true
I20260618 22:39:26.876473 14004 raft_consensus.cc:2454] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c749157c20c458bb3c63cd5b01247b9 in term 0.
I20260618 22:39:26.876696 13726 raft_consensus.cc:2454] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c749157c20c458bb3c63cd5b01247b9 in term 0.
I20260618 22:39:26.877169 13798 leader_election.cc:302] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d7cdbc2b91b4a5ba6f70c4605d8cc23, 2c749157c20c458bb3c63cd5b01247b9; no voters:
I20260618 22:39:26.877548 14086 raft_consensus.cc:2781] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:39:26.877691 14086 raft_consensus.cc:482] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:39:26.877784 14086 raft_consensus.cc:3037] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:26.879920 14086 raft_consensus.cc:504] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } }
I20260618 22:39:26.880470 14086 leader_election.cc:288] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [CANDIDATE]: Term 1 election: Requested vote from peers 1bdb809d25fe458b98b389dfd4cb2c3b (127.5.213.193:40341), 1d7cdbc2b91b4a5ba6f70c4605d8cc23 (127.5.213.195:37043)
I20260618 22:39:26.880995 13726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "826419e1a9c74db3af5025a13f9af266" candidate_uuid: "2c749157c20c458bb3c63cd5b01247b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b"
I20260618 22:39:26.880995 14004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "826419e1a9c74db3af5025a13f9af266" candidate_uuid: "2c749157c20c458bb3c63cd5b01247b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23"
I20260618 22:39:26.881202 14004 raft_consensus.cc:3037] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:26.881211 13726 raft_consensus.cc:3037] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:26.884261 14004 raft_consensus.cc:2454] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c749157c20c458bb3c63cd5b01247b9 in term 1.
I20260618 22:39:26.884261 13726 raft_consensus.cc:2454] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c749157c20c458bb3c63cd5b01247b9 in term 1.
I20260618 22:39:26.884729 13796 leader_election.cc:302] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1bdb809d25fe458b98b389dfd4cb2c3b, 1d7cdbc2b91b4a5ba6f70c4605d8cc23, 2c749157c20c458bb3c63cd5b01247b9; no voters:
I20260618 22:39:26.885048 14086 raft_consensus.cc:2781] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:26.885457 14086 raft_consensus.cc:686] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [term 1 LEADER]: Becoming Leader. State: Replica: 2c749157c20c458bb3c63cd5b01247b9, State: Running, Role: LEADER
I20260618 22:39:26.885982 14086 consensus_queue.cc:237] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } }
I20260618 22:39:26.890455 13569 catalog_manager.cc:5495] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c749157c20c458bb3c63cd5b01247b9 (127.5.213.194). New cstate: current_term: 1 leader_uuid: "2c749157c20c458bb3c63cd5b01247b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } health_report { overall_health: HEALTHY } } }
I20260618 22:39:26.925979 14077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:26.920940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54302 (local address 127.5.213.194:35219)
0618 22:39:26.921110 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:26.921114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:26.921131 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:39:26.921276 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:26.921280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:26.921387 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:26.921525 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:39:26.921865 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.922595 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.923929 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:26.924342 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:26.924435 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:26.924873 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:39:26.925419 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:39:26.925536 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:26.925621 (+ 85us) server_negotiation.cc:299] Negotiation successful
0618 22:39:26.925690 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:39:26.933974 13726 raft_consensus.cc:1261] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 FOLLOWER]: Refusing update from remote peer 2c749157c20c458bb3c63cd5b01247b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:39:26.933991 14004 raft_consensus.cc:1261] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 1 FOLLOWER]: Refusing update from remote peer 2c749157c20c458bb3c63cd5b01247b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:39:26.934907 14086 consensus_queue.cc:1035] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:26.935173 14090 consensus_queue.cc:1035] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:26.945329 14093 mvcc.cc:205] Tried to move back new op lower bound from 7298344414952644608 to 7298344414768693248. Current Snapshot: MvccSnapshot[applied={T|T < 7298344414952644608}]
I20260618 22:39:26.946794 14095 mvcc.cc:205] Tried to move back new op lower bound from 7298344414952644608 to 7298344414768693248. Current Snapshot: MvccSnapshot[applied={T|T < 7298344414952644608}]
I20260618 22:39:26.959966 13568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:39:26.960325 13568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:39:26.963846 13568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca: Sending DeleteTablet for 3 replicas of tablet 826419e1a9c74db3af5025a13f9af266
I20260618 22:39:26.965010 13984 tablet_service.cc:1514] Processing DeleteTablet for tablet 826419e1a9c74db3af5025a13f9af266 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:26 UTC) from {username='oryx', principal='oryx/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:43116
I20260618 22:39:26.965121 13706 tablet_service.cc:1514] Processing DeleteTablet for tablet 826419e1a9c74db3af5025a13f9af266 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:26 UTC) from {username='oryx', principal='oryx/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:43510
I20260618 22:39:26.965939 13845 tablet_service.cc:1514] Processing DeleteTablet for tablet 826419e1a9c74db3af5025a13f9af266 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:26 UTC) from {username='oryx', principal='oryx/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:54286
I20260618 22:39:26.966795 14105 tablet_replica.cc:330] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9: stopping tablet replica
I20260618 22:39:26.966794 13568 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260618 22:39:26.967260 14105 raft_consensus.cc:2229] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:39:26.967940 14105 raft_consensus.cc:2258] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260618 22:39:26.968217 13568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:39:26.969947 14105 ts_tablet_manager.cc:1905] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:26.976717 13705 tablet_service.cc:1467] Processing CreateTablet for tablet e0f1dd13811a4dd9bee64edd7e6b8d96 (DEFAULT_TABLE table=test-table [id=530be68123f84b669979ca39042b3975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:26.977065 13983 tablet_service.cc:1467] Processing CreateTablet for tablet e0f1dd13811a4dd9bee64edd7e6b8d96 (DEFAULT_TABLE table=test-table [id=530be68123f84b669979ca39042b3975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:26.977177 14105 ts_tablet_manager.cc:1918] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:39:26.977258 13705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f1dd13811a4dd9bee64edd7e6b8d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:26.977339 14105 log.cc:1196] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/826419e1a9c74db3af5025a13f9af266
I20260618 22:39:26.977567 13983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f1dd13811a4dd9bee64edd7e6b8d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:26.977825 14105 ts_tablet_manager.cc:1939] T 826419e1a9c74db3af5025a13f9af266 P 2c749157c20c458bb3c63cd5b01247b9: Deleting consensus metadata
I20260618 22:39:26.982256 14080 tablet_bootstrap.cc:492] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Bootstrap starting.
I20260618 22:39:26.982987 13553 catalog_manager.cc:4841] TS 2c749157c20c458bb3c63cd5b01247b9 (127.5.213.194:35219): tablet 826419e1a9c74db3af5025a13f9af266 (table test-table [id=9a3e32727a0d4e198d4ec552d1ec5fba]) successfully deleted
I20260618 22:39:26.983771 14080 tablet_bootstrap.cc:654] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:26.984413 14081 tablet_bootstrap.cc:492] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b: Bootstrap starting.
I20260618 22:39:26.984685 13845 tablet_service.cc:1467] Processing CreateTablet for tablet e0f1dd13811a4dd9bee64edd7e6b8d96 (DEFAULT_TABLE table=test-table [id=530be68123f84b669979ca39042b3975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:26.985140 13845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f1dd13811a4dd9bee64edd7e6b8d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:26.985893 14081 tablet_bootstrap.cc:654] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:26.993150 14103 tablet_replica.cc:330] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: stopping tablet replica
I20260618 22:39:26.993696 14103 raft_consensus.cc:2229] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:39:26.993924 14081 tablet_bootstrap.cc:492] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b: No bootstrap required, opened a new log
I20260618 22:39:26.994071 14081 ts_tablet_manager.cc:1397] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260618 22:39:26.994344 14103 raft_consensus.cc:2258] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:26.994859 14081 raft_consensus.cc:348] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:26.995061 14081 raft_consensus.cc:374] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:26.995134 14081 raft_consensus.cc:729] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bdb809d25fe458b98b389dfd4cb2c3b, State: Initialized, Role: FOLLOWER
I20260618 22:39:26.995688 14079 tablet_bootstrap.cc:492] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9: Bootstrap starting.
I20260618 22:39:26.996902 14103 ts_tablet_manager.cc:1905] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:26.997329 14079 tablet_bootstrap.cc:654] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:26.995352 14081 consensus_queue.cc:260] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.001832 14081 ts_tablet_manager.cc:1428] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260618 22:39:27.003544 14103 ts_tablet_manager.cc:1918] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:39:27.003744 14103 log.cc:1196] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/826419e1a9c74db3af5025a13f9af266
I20260618 22:39:27.004292 14103 ts_tablet_manager.cc:1939] T 826419e1a9c74db3af5025a13f9af266 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Deleting consensus metadata
I20260618 22:39:27.006081 13555 catalog_manager.cc:4841] TS 1d7cdbc2b91b4a5ba6f70c4605d8cc23 (127.5.213.195:37043): tablet 826419e1a9c74db3af5025a13f9af266 (table test-table [id=9a3e32727a0d4e198d4ec552d1ec5fba]) successfully deleted
I20260618 22:39:27.007186 14104 tablet_replica.cc:330] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b: stopping tablet replica
I20260618 22:39:27.007683 14104 raft_consensus.cc:2229] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:39:27.008212 14104 raft_consensus.cc:2258] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:27.008278 14079 tablet_bootstrap.cc:492] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9: No bootstrap required, opened a new log
I20260618 22:39:27.008427 14079 ts_tablet_manager.cc:1397] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20260618 22:39:27.009323 14080 tablet_bootstrap.cc:492] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: No bootstrap required, opened a new log
I20260618 22:39:27.009310 14079 raft_consensus.cc:348] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.009634 14079 raft_consensus.cc:374] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:27.009788 14079 raft_consensus.cc:729] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c749157c20c458bb3c63cd5b01247b9, State: Initialized, Role: FOLLOWER
I20260618 22:39:27.009847 14080 ts_tablet_manager.cc:1397] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.002s
I20260618 22:39:27.010154 14079 consensus_queue.cc:260] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.010605 14080 raft_consensus.cc:348] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.010851 14080 raft_consensus.cc:374] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:27.010938 14104 ts_tablet_manager.cc:1905] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:27.010962 14080 raft_consensus.cc:729] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d7cdbc2b91b4a5ba6f70c4605d8cc23, State: Initialized, Role: FOLLOWER
I20260618 22:39:27.011215 14080 consensus_queue.cc:260] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.013408 14079 ts_tablet_manager.cc:1428] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260618 22:39:27.013592 14080 ts_tablet_manager.cc:1428] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20260618 22:39:27.017997 14104 ts_tablet_manager.cc:1918] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:39:27.018163 14104 log.cc:1196] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/826419e1a9c74db3af5025a13f9af266
I20260618 22:39:27.018672 14104 ts_tablet_manager.cc:1939] T 826419e1a9c74db3af5025a13f9af266 P 1bdb809d25fe458b98b389dfd4cb2c3b: Deleting consensus metadata
I20260618 22:39:27.020464 13553 catalog_manager.cc:4841] TS 1bdb809d25fe458b98b389dfd4cb2c3b (127.5.213.193:40341): tablet 826419e1a9c74db3af5025a13f9af266 (table test-table [id=9a3e32727a0d4e198d4ec552d1ec5fba]) successfully deleted
W20260618 22:39:27.211841 14055 tablet.cc:2369] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:39:27.238701 13916 tablet.cc:2369] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:39:27.239861 13777 tablet.cc:2369] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:39:27.291381 14085 raft_consensus.cc:482] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:39:27.291585 14085 raft_consensus.cc:504] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.293145 14085 leader_election.cc:288] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c749157c20c458bb3c63cd5b01247b9 (127.5.213.194:35219), 1d7cdbc2b91b4a5ba6f70c4605d8cc23 (127.5.213.195:37043)
I20260618 22:39:27.300420 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:27.293867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:33995 (local address 127.5.213.195:37043)
0618 22:39:27.294039 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:27.294045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:27.294456 (+ 411us) server_negotiation.cc:407] Connection header received
0618 22:39:27.294589 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:27.294594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:27.294698 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:27.294795 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:27.295459 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.296483 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.298341 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.299395 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.299557 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:27.299634 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:27.299971 (+ 337us) server_negotiation.cc:299] Negotiation successful
0618 22:39:27.300136 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:39:27.300509 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:27.293475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:37043 (local address 127.5.213.193:33995)
0618 22:39:27.294364 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:27.294378 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:27.294474 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:27.294827 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:27.294832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:27.294858 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:27.295282 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:27.295294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.296595 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.296600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:27.298189 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:27.298202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.299614 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.299621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:27.299803 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:27.299839 (+ 36us) client_negotiation.cc:769] Sending connection context
0618 22:39:27.299944 (+ 105us) client_negotiation.cc:240] Negotiation successful
0618 22:39:27.300093 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":33,"thread_start_us":575,"threads_started":1}
I20260618 22:39:27.301278 14077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:27.293589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:53953 (local address 127.5.213.194:35219)
0618 22:39:27.293752 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:27.293758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:27.294129 (+ 371us) server_negotiation.cc:407] Connection header received
0618 22:39:27.294426 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:27.294433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:27.294513 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:27.294612 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:27.295216 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.296402 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.298946 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.300053 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.300220 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:27.300289 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:27.300890 (+ 601us) server_negotiation.cc:299] Negotiation successful
0618 22:39:27.301027 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 22:39:27.301510 14004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f1dd13811a4dd9bee64edd7e6b8d96" candidate_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" is_pre_election: true
I20260618 22:39:27.301765 14004 raft_consensus.cc:2454] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bdb809d25fe458b98b389dfd4cb2c3b in term 0.
I20260618 22:39:27.301976 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:27.293375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:35219 (local address 127.5.213.193:53953)
0618 22:39:27.294009 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:27.294032 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:27.294250 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:27.294651 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:27.294656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:27.294677 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:27.295065 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:27.295075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.296562 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.296568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:27.298784 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:27.298798 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.300467 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.300473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:27.300663 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:27.300729 (+ 66us) client_negotiation.cc:769] Sending connection context
0618 22:39:27.301488 (+ 759us) client_negotiation.cc:240] Negotiation successful
0618 22:39:27.301622 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":3,"thread_start_us":389,"threads_started":1}
I20260618 22:39:27.302238 13658 leader_election.cc:302] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bdb809d25fe458b98b389dfd4cb2c3b, 1d7cdbc2b91b4a5ba6f70c4605d8cc23; no voters:
I20260618 22:39:27.302616 14085 raft_consensus.cc:2781] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:39:27.302755 14085 raft_consensus.cc:482] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:39:27.302853 14085 raft_consensus.cc:3037] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:27.303213 13870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f1dd13811a4dd9bee64edd7e6b8d96" candidate_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c749157c20c458bb3c63cd5b01247b9" is_pre_election: true
I20260618 22:39:27.303596 13870 raft_consensus.cc:2454] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bdb809d25fe458b98b389dfd4cb2c3b in term 0.
I20260618 22:39:27.305064 14085 raft_consensus.cc:504] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.305657 14085 leader_election.cc:288] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [CANDIDATE]: Term 1 election: Requested vote from peers 2c749157c20c458bb3c63cd5b01247b9 (127.5.213.194:35219), 1d7cdbc2b91b4a5ba6f70c4605d8cc23 (127.5.213.195:37043)
I20260618 22:39:27.306084 13870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f1dd13811a4dd9bee64edd7e6b8d96" candidate_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c749157c20c458bb3c63cd5b01247b9"
I20260618 22:39:27.306106 14004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f1dd13811a4dd9bee64edd7e6b8d96" candidate_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23"
I20260618 22:39:27.306262 13870 raft_consensus.cc:3037] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:27.306308 14004 raft_consensus.cc:3037] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:27.308461 13870 raft_consensus.cc:2454] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bdb809d25fe458b98b389dfd4cb2c3b in term 1.
I20260618 22:39:27.308830 13656 leader_election.cc:302] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bdb809d25fe458b98b389dfd4cb2c3b, 2c749157c20c458bb3c63cd5b01247b9; no voters:
I20260618 22:39:27.309073 14085 raft_consensus.cc:2781] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:27.309110 14004 raft_consensus.cc:2454] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bdb809d25fe458b98b389dfd4cb2c3b in term 1.
I20260618 22:39:27.309479 14085 raft_consensus.cc:686] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 LEADER]: Becoming Leader. State: Replica: 1bdb809d25fe458b98b389dfd4cb2c3b, State: Running, Role: LEADER
I20260618 22:39:27.310022 14085 consensus_queue.cc:237] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.313403 13569 catalog_manager.cc:5495] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bdb809d25fe458b98b389dfd4cb2c3b (127.5.213.193). New cstate: current_term: 1 leader_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } health_report { overall_health: UNKNOWN } } }
Jun 18 22:39:27 dist-test-slave-lnj3 krb5kdc[13524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822364, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.254@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM
I20260618 22:39:27.405383 14118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:27.392765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:36989 (local address 127.0.0.1:46564)
0618 22:39:27.393444 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:27.393464 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:27.393596 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:27.393874 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:27.393877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:27.394478 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:27.394700 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:27.394709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.395756 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.395759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:27.396693 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:27.396704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.397434 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.397438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:27.397558 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:27.398173 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:27.398191 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:27.400489 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:27.403236 (+ 2747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:27.403245 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:27.403251 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:27.403640 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:27.403936 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:27.403939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:27.403941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:27.404068 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:27.404568 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:27.404575 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:27.404718 (+ 143us) client_negotiation.cc:769] Sending connection context
0618 22:39:27.404859 (+ 141us) client_negotiation.cc:240] Negotiation successful
0618 22:39:27.405052 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":324,"threads_started":1}
I20260618 22:39:27.405469 14119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:27.392759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.5.213.254:36989)
0618 22:39:27.393607 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:27.393611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:27.393631 (+ 20us) server_negotiation.cc:407] Connection header received
0618 22:39:27.393683 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:27.393687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:27.393739 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:27.393830 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:27.394845 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.395630 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.396830 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.397215 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.397311 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:27.400692 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:27.400697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:27.400700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:27.400727 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:27.403134 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:27.403736 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:27.403740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:27.403742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:27.403852 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:27.404187 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:27.404191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:27.404193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:27.404463 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:27.404577 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:27.404947 (+ 370us) server_negotiation.cc:299] Negotiation successful
0618 22:39:27.405165 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":739,"thread_start_us":615,"threads_started":1}
I20260618 22:39:27.410156 13569 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:46564:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260618 22:39:27.411142 13569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:39:27.417316 13706 tablet_service.cc:1467] Processing CreateTablet for tablet 5837926976c741098b97e528ed7ac089 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c54a883cc0ed4bd89ea5a3e4faf23194]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:39:27.417481 13845 tablet_service.cc:1467] Processing CreateTablet for tablet 5837926976c741098b97e528ed7ac089 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c54a883cc0ed4bd89ea5a3e4faf23194]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:39:27.417726 13984 tablet_service.cc:1467] Processing CreateTablet for tablet 5837926976c741098b97e528ed7ac089 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c54a883cc0ed4bd89ea5a3e4faf23194]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:39:27.417910 13845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5837926976c741098b97e528ed7ac089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:27.417904 13706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5837926976c741098b97e528ed7ac089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:27.418171 13984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5837926976c741098b97e528ed7ac089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:27.423123 14080 tablet_bootstrap.cc:492] T 5837926976c741098b97e528ed7ac089 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Bootstrap starting.
I20260618 22:39:27.424439 14080 tablet_bootstrap.cc:654] T 5837926976c741098b97e528ed7ac089 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:27.424535 14079 tablet_bootstrap.cc:492] T 5837926976c741098b97e528ed7ac089 P 2c749157c20c458bb3c63cd5b01247b9: Bootstrap starting.
I20260618 22:39:27.424556 14081 tablet_bootstrap.cc:492] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b: Bootstrap starting.
I20260618 22:39:27.425850 14079 tablet_bootstrap.cc:654] T 5837926976c741098b97e528ed7ac089 P 2c749157c20c458bb3c63cd5b01247b9: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:27.426008 14081 tablet_bootstrap.cc:654] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:27.426999 14080 tablet_bootstrap.cc:492] T 5837926976c741098b97e528ed7ac089 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: No bootstrap required, opened a new log
I20260618 22:39:27.427155 14080 ts_tablet_manager.cc:1397] T 5837926976c741098b97e528ed7ac089 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260618 22:39:27.427731 14079 tablet_bootstrap.cc:492] T 5837926976c741098b97e528ed7ac089 P 2c749157c20c458bb3c63cd5b01247b9: No bootstrap required, opened a new log
I20260618 22:39:27.427723 14080 raft_consensus.cc:348] T 5837926976c741098b97e528ed7ac089 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.427876 14079 ts_tablet_manager.cc:1397] T 5837926976c741098b97e528ed7ac089 P 2c749157c20c458bb3c63cd5b01247b9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260618 22:39:27.427953 14080 raft_consensus.cc:374] T 5837926976c741098b97e528ed7ac089 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:27.428035 14080 raft_consensus.cc:729] T 5837926976c741098b97e528ed7ac089 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d7cdbc2b91b4a5ba6f70c4605d8cc23, State: Initialized, Role: FOLLOWER
I20260618 22:39:27.428189 14081 tablet_bootstrap.cc:492] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b: No bootstrap required, opened a new log
I20260618 22:39:27.428329 14081 ts_tablet_manager.cc:1397] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260618 22:39:27.428265 14080 consensus_queue.cc:260] T 5837926976c741098b97e528ed7ac089 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.428478 14079 raft_consensus.cc:348] T 5837926976c741098b97e528ed7ac089 P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.428727 14079 raft_consensus.cc:374] T 5837926976c741098b97e528ed7ac089 P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:27.428807 14079 raft_consensus.cc:729] T 5837926976c741098b97e528ed7ac089 P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c749157c20c458bb3c63cd5b01247b9, State: Initialized, Role: FOLLOWER
I20260618 22:39:27.428862 14080 ts_tablet_manager.cc:1428] T 5837926976c741098b97e528ed7ac089 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260618 22:39:27.428720 14087 tablet_replica.cc:439] T 5837926976c741098b97e528ed7ac089 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } } }
I20260618 22:39:27.428954 14081 raft_consensus.cc:348] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.429117 14087 tablet_replica.cc:442] T 5837926976c741098b97e528ed7ac089 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:39:27.429013 14079 consensus_queue.cc:260] T 5837926976c741098b97e528ed7ac089 P 2c749157c20c458bb3c63cd5b01247b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.429168 14081 raft_consensus.cc:374] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:27.429234 14081 raft_consensus.cc:729] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bdb809d25fe458b98b389dfd4cb2c3b, State: Initialized, Role: FOLLOWER
I20260618 22:39:27.429477 14079 ts_tablet_manager.cc:1428] T 5837926976c741098b97e528ed7ac089 P 2c749157c20c458bb3c63cd5b01247b9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260618 22:39:27.429459 14081 consensus_queue.cc:260] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.429796 14085 tablet_replica.cc:439] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } } }
I20260618 22:39:27.429816 14098 tablet_replica.cc:439] T 5837926976c741098b97e528ed7ac089 P 2c749157c20c458bb3c63cd5b01247b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } } }
I20260618 22:39:27.430040 14085 tablet_replica.cc:442] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:39:27.430049 14098 tablet_replica.cc:442] T 5837926976c741098b97e528ed7ac089 P 2c749157c20c458bb3c63cd5b01247b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:39:27.430962 14081 ts_tablet_manager.cc:1428] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260618 22:39:27.489959 14085 raft_consensus.cc:482] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:39:27.490156 14085 raft_consensus.cc:504] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.490762 14085 leader_election.cc:288] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c749157c20c458bb3c63cd5b01247b9 (127.5.213.194:35219), 1d7cdbc2b91b4a5ba6f70c4605d8cc23 (127.5.213.195:37043)
I20260618 22:39:27.491217 13870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5837926976c741098b97e528ed7ac089" candidate_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c749157c20c458bb3c63cd5b01247b9" is_pre_election: true
I20260618 22:39:27.491285 14004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5837926976c741098b97e528ed7ac089" candidate_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" is_pre_election: true
I20260618 22:39:27.491467 13870 raft_consensus.cc:2454] T 5837926976c741098b97e528ed7ac089 P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bdb809d25fe458b98b389dfd4cb2c3b in term 0.
I20260618 22:39:27.491513 14004 raft_consensus.cc:2454] T 5837926976c741098b97e528ed7ac089 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bdb809d25fe458b98b389dfd4cb2c3b in term 0.
I20260618 22:39:27.491888 13658 leader_election.cc:302] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bdb809d25fe458b98b389dfd4cb2c3b, 1d7cdbc2b91b4a5ba6f70c4605d8cc23; no voters:
I20260618 22:39:27.492184 14085 raft_consensus.cc:2781] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:39:27.492280 14085 raft_consensus.cc:482] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:39:27.492364 14085 raft_consensus.cc:3037] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:27.494176 14085 raft_consensus.cc:504] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.494666 14085 leader_election.cc:288] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b [CANDIDATE]: Term 1 election: Requested vote from peers 2c749157c20c458bb3c63cd5b01247b9 (127.5.213.194:35219), 1d7cdbc2b91b4a5ba6f70c4605d8cc23 (127.5.213.195:37043)
I20260618 22:39:27.495052 13870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5837926976c741098b97e528ed7ac089" candidate_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c749157c20c458bb3c63cd5b01247b9"
I20260618 22:39:27.495097 14004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5837926976c741098b97e528ed7ac089" candidate_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23"
I20260618 22:39:27.495232 13870 raft_consensus.cc:3037] T 5837926976c741098b97e528ed7ac089 P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:27.495278 14004 raft_consensus.cc:3037] T 5837926976c741098b97e528ed7ac089 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:27.497021 13870 raft_consensus.cc:2454] T 5837926976c741098b97e528ed7ac089 P 2c749157c20c458bb3c63cd5b01247b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bdb809d25fe458b98b389dfd4cb2c3b in term 1.
I20260618 22:39:27.497026 14004 raft_consensus.cc:2454] T 5837926976c741098b97e528ed7ac089 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bdb809d25fe458b98b389dfd4cb2c3b in term 1.
I20260618 22:39:27.497411 13658 leader_election.cc:302] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bdb809d25fe458b98b389dfd4cb2c3b, 1d7cdbc2b91b4a5ba6f70c4605d8cc23; no voters:
I20260618 22:39:27.497638 14085 raft_consensus.cc:2781] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:27.497798 14085 raft_consensus.cc:686] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 LEADER]: Becoming Leader. State: Replica: 1bdb809d25fe458b98b389dfd4cb2c3b, State: Running, Role: LEADER
I20260618 22:39:27.498039 14085 consensus_queue.cc:237] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.499590 14111 tablet_replica.cc:439] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b: TxnStatusTablet state changed. Reason: New leader 1bdb809d25fe458b98b389dfd4cb2c3b. Latest consensus state: current_term: 1 leader_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } } }
I20260618 22:39:27.499884 14111 tablet_replica.cc:442] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:39:27.500433 14124 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260618 22:39:27.500650 14124 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260618 22:39:27.500874 13566 catalog_manager.cc:5495] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bdb809d25fe458b98b389dfd4cb2c3b (127.5.213.193). New cstate: current_term: 1 leader_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } health_report { overall_health: UNKNOWN } } }
I20260618 22:39:27.524322 14118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:27.512355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:40341 (local address 127.0.0.1:43518)
0618 22:39:27.512590 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:27.512610 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:27.512746 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:27.519472 (+ 6726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:27.519477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:27.519491 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:39:27.519680 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:27.519689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.520880 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.520886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:27.521865 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:27.521877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.522662 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.522668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:27.522829 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:27.522890 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:39:27.523783 (+ 893us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:39:27.523807 (+ 24us) client_negotiation.cc:769] Sending connection context
0618 22:39:27.523878 (+ 71us) client_negotiation.cc:240] Negotiation successful
0618 22:39:27.524030 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260618 22:39:27.524330 14127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:27.512556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.5.213.193:40341)
0618 22:39:27.519057 (+ 6501us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:27.519062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:27.519087 (+ 25us) server_negotiation.cc:407] Connection header received
0618 22:39:27.519173 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:27.519176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:27.519262 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:27.519420 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:39:27.519837 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.520749 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.522004 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.522492 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.522602 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:27.523085 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:39:27.523677 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:39:27.523841 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:27.523893 (+ 52us) server_negotiation.cc:299] Negotiation successful
0618 22:39:27.523969 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6384,"thread_start_us":6262,"threads_started":1}
I20260618 22:39:27.528680 14004 raft_consensus.cc:1261] T 5837926976c741098b97e528ed7ac089 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 1 FOLLOWER]: Refusing update from remote peer 1bdb809d25fe458b98b389dfd4cb2c3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:39:27.528816 13870 raft_consensus.cc:1261] T 5837926976c741098b97e528ed7ac089 P 2c749157c20c458bb3c63cd5b01247b9 [term 1 FOLLOWER]: Refusing update from remote peer 1bdb809d25fe458b98b389dfd4cb2c3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:39:27.529457 14085 consensus_queue.cc:1035] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:27.529812 14111 consensus_queue.cc:1035] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:27.533298 14087 tablet_replica.cc:439] T 5837926976c741098b97e528ed7ac089 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: TxnStatusTablet state changed. Reason: New leader 1bdb809d25fe458b98b389dfd4cb2c3b. Latest consensus state: current_term: 1 leader_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } } }
I20260618 22:39:27.533635 14087 tablet_replica.cc:442] T 5837926976c741098b97e528ed7ac089 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:39:27.536266 14129 mvcc.cc:205] Tried to move back new op lower bound from 7298344417389944832 to 7298344417273896960. Current Snapshot: MvccSnapshot[applied={T|T < 7298344417389944832}]
I20260618 22:39:27.537122 14085 tablet_replica.cc:439] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } } }
I20260618 22:39:27.537453 14085 tablet_replica.cc:442] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:39:27.539072 14087 tablet_replica.cc:439] T 5837926976c741098b97e528ed7ac089 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } } }
I20260618 22:39:27.539291 14087 tablet_replica.cc:442] T 5837926976c741098b97e528ed7ac089 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:39:27.544082 14098 tablet_replica.cc:439] T 5837926976c741098b97e528ed7ac089 P 2c749157c20c458bb3c63cd5b01247b9: TxnStatusTablet state changed. Reason: New leader 1bdb809d25fe458b98b389dfd4cb2c3b. Latest consensus state: current_term: 1 leader_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } } }
I20260618 22:39:27.544404 14098 tablet_replica.cc:442] T 5837926976c741098b97e528ed7ac089 P 2c749157c20c458bb3c63cd5b01247b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:39:27.545917 14111 tablet_replica.cc:439] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } } }
I20260618 22:39:27.546202 14111 tablet_replica.cc:442] T 5837926976c741098b97e528ed7ac089 P 1bdb809d25fe458b98b389dfd4cb2c3b: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:39:27.548969 14098 tablet_replica.cc:439] T 5837926976c741098b97e528ed7ac089 P 2c749157c20c458bb3c63cd5b01247b9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } } }
I20260618 22:39:27.549214 14098 tablet_replica.cc:442] T 5837926976c741098b97e528ed7ac089 P 2c749157c20c458bb3c63cd5b01247b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:39:27.557683 14127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:27.552463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43524 (local address 127.5.213.193:40341)
0618 22:39:27.552635 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:27.552642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:27.553020 (+ 378us) server_negotiation.cc:407] Connection header received
0618 22:39:27.553190 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:27.553196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:27.553264 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:27.553348 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:39:27.553772 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.554483 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.555873 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.556377 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.556483 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:27.556885 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:39:27.556993 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:39:27.557074 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:27.557266 (+ 192us) server_negotiation.cc:299] Negotiation successful
0618 22:39:27.557353 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 22:39:27.573359 14127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:27.567964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:39305 (local address 127.5.213.193:40341)
0618 22:39:27.568108 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:27.568114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:27.568566 (+ 452us) server_negotiation.cc:407] Connection header received
0618 22:39:27.568757 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:27.568764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:27.568830 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:27.568899 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:39:27.569451 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.570221 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.571684 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.572096 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.572196 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:27.572637 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:39:27.572752 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:39:27.572832 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:27.572948 (+ 116us) server_negotiation.cc:299] Negotiation successful
0618 22:39:27.573011 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 22:39:27.573451 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:27.567925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:40341 (local address 127.5.213.193:39305)
0618 22:39:27.568478 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:27.568493 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:27.568626 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:27.568923 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:27.568927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:27.568937 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:39:27.569307 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:27.569319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.570342 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.570347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:27.571553 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:27.571564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.572265 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.572278 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:27.572395 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:27.572441 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:39:27.572843 (+ 402us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:39:27.572852 (+ 9us) client_negotiation.cc:769] Sending connection context
0618 22:39:27.572923 (+ 71us) client_negotiation.cc:240] Negotiation successful
0618 22:39:27.573045 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":303,"threads_started":1}
I20260618 22:39:27.582621 14004 raft_consensus.cc:1261] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 1 FOLLOWER]: Refusing update from remote peer 1bdb809d25fe458b98b389dfd4cb2c3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:39:27.582744 13870 raft_consensus.cc:1261] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9 [term 1 FOLLOWER]: Refusing update from remote peer 1bdb809d25fe458b98b389dfd4cb2c3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:39:27.583216 14085 consensus_queue.cc:1035] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:27.583541 14111 consensus_queue.cc:1035] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:27.657645 13567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:39:27.657913 13567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:39:27.661420 13567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca: Sending DeleteTablet for 3 replicas of tablet e0f1dd13811a4dd9bee64edd7e6b8d96
I20260618 22:39:27.662247 13984 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f1dd13811a4dd9bee64edd7e6b8d96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:27 UTC) from {username='oryx', principal='oryx/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:43116
I20260618 22:39:27.662283 13845 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f1dd13811a4dd9bee64edd7e6b8d96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:27 UTC) from {username='oryx', principal='oryx/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:54286
I20260618 22:39:27.663204 13706 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f1dd13811a4dd9bee64edd7e6b8d96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:27 UTC) from {username='oryx', principal='oryx/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:43510
I20260618 22:39:27.671072 14148 tablet_replica.cc:330] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b: stopping tablet replica
I20260618 22:39:27.671332 14148 raft_consensus.cc:2229] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:39:27.672044 14148 raft_consensus.cc:2258] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:27.673344 14148 ts_tablet_manager.cc:1905] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:27.678133 14146 tablet_replica.cc:330] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: stopping tablet replica
I20260618 22:39:27.678324 14146 raft_consensus.cc:2229] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:39:27.678587 14146 raft_consensus.cc:2258] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:27.679515 14146 ts_tablet_manager.cc:1905] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:27.680575 14148 ts_tablet_manager.cc:1918] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:39:27.680707 14148 log.cc:1196] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/e0f1dd13811a4dd9bee64edd7e6b8d96
I20260618 22:39:27.681133 14148 ts_tablet_manager.cc:1939] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1bdb809d25fe458b98b389dfd4cb2c3b: Deleting consensus metadata
I20260618 22:39:27.682350 13553 catalog_manager.cc:4841] TS 1bdb809d25fe458b98b389dfd4cb2c3b (127.5.213.193:40341): tablet e0f1dd13811a4dd9bee64edd7e6b8d96 (table test-table [id=530be68123f84b669979ca39042b3975]) successfully deleted
I20260618 22:39:27.683092 14147 tablet_replica.cc:330] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9: stopping tablet replica
I20260618 22:39:27.683347 14147 raft_consensus.cc:2229] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:39:27.683624 14147 raft_consensus.cc:2258] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:27.684603 14147 ts_tablet_manager.cc:1905] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:27.685688 14146 ts_tablet_manager.cc:1918] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:39:27.685802 14146 log.cc:1196] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/e0f1dd13811a4dd9bee64edd7e6b8d96
I20260618 22:39:27.686213 14146 ts_tablet_manager.cc:1939] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Deleting consensus metadata
I20260618 22:39:27.687325 13555 catalog_manager.cc:4841] TS 1d7cdbc2b91b4a5ba6f70c4605d8cc23 (127.5.213.195:37043): tablet e0f1dd13811a4dd9bee64edd7e6b8d96 (table test-table [id=530be68123f84b669979ca39042b3975]) successfully deleted
I20260618 22:39:27.690197 14147 ts_tablet_manager.cc:1918] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:39:27.690307 14147 log.cc:1196] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/e0f1dd13811a4dd9bee64edd7e6b8d96
I20260618 22:39:27.690616 14147 ts_tablet_manager.cc:1939] T e0f1dd13811a4dd9bee64edd7e6b8d96 P 2c749157c20c458bb3c63cd5b01247b9: Deleting consensus metadata
I20260618 22:39:27.691484 13553 catalog_manager.cc:4841] TS 2c749157c20c458bb3c63cd5b01247b9 (127.5.213.194:35219): tablet e0f1dd13811a4dd9bee64edd7e6b8d96 (table test-table [id=530be68123f84b669979ca39042b3975]) successfully deleted
Jun 18 22:39:27 dist-test-slave-lnj3 krb5kdc[13524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 22:39:27 dist-test-slave-lnj3 krb5kdc[13524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:27 dist-test-slave-lnj3 krb5kdc[13524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
W20260618 22:39:27.727465 14119 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:27.714102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.5.213.254:36989)
0618 22:39:27.714292 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:27.714298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:27.714801 (+ 503us) server_negotiation.cc:407] Connection header received
0618 22:39:27.714961 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:27.714967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:27.715065 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:27.715177 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:27.716747 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.717889 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.719749 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.720453 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.720592 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:27.727175 (+ 6583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46568: BlockingRecv error: recv got EOF from 127.0.0.1:46568 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 18 22:39:27 dist-test-slave-lnj3 krb5kdc[13524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM
I20260618 22:39:27.755899 14119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:27.739984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46576 (local address 127.5.213.254:36989)
0618 22:39:27.740182 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:27.740189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:27.743192 (+ 3003us) server_negotiation.cc:407] Connection header received
0618 22:39:27.743373 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:27.743378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:27.743442 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:27.743523 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:27.744745 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.745475 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.746711 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.747126 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.747237 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:27.750937 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:27.750942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:27.750946 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:27.750978 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:27.753414 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:27.754008 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:27.754012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:27.754014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:27.754143 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:27.754496 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:27.754499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:27.754501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:27.754802 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:27.755041 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:27.755308 (+ 267us) server_negotiation.cc:299] Negotiation successful
0618 22:39:27.755553 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:39:27.759363 13567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:39:27.759994 13567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:39:27.766654 13706 tablet_service.cc:1467] Processing CreateTablet for tablet cf7d2608be714be9bbc599f270d4b01a (DEFAULT_TABLE table=test-table [id=ae6ab3abefd240ea8119a20da66d299a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:27.767004 13845 tablet_service.cc:1467] Processing CreateTablet for tablet cf7d2608be714be9bbc599f270d4b01a (DEFAULT_TABLE table=test-table [id=ae6ab3abefd240ea8119a20da66d299a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:27.767179 13706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7d2608be714be9bbc599f270d4b01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:27.767375 13845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7d2608be714be9bbc599f270d4b01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:27.771018 13984 tablet_service.cc:1467] Processing CreateTablet for tablet cf7d2608be714be9bbc599f270d4b01a (DEFAULT_TABLE table=test-table [id=ae6ab3abefd240ea8119a20da66d299a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:27.771410 13984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7d2608be714be9bbc599f270d4b01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:27.773921 14081 tablet_bootstrap.cc:492] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b: Bootstrap starting.
I20260618 22:39:27.775569 14081 tablet_bootstrap.cc:654] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:27.779685 14079 tablet_bootstrap.cc:492] T cf7d2608be714be9bbc599f270d4b01a P 2c749157c20c458bb3c63cd5b01247b9: Bootstrap starting.
I20260618 22:39:27.781154 14079 tablet_bootstrap.cc:654] T cf7d2608be714be9bbc599f270d4b01a P 2c749157c20c458bb3c63cd5b01247b9: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:27.783856 14080 tablet_bootstrap.cc:492] T cf7d2608be714be9bbc599f270d4b01a P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Bootstrap starting.
I20260618 22:39:27.785228 14081 tablet_bootstrap.cc:492] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b: No bootstrap required, opened a new log
I20260618 22:39:27.785351 14080 tablet_bootstrap.cc:654] T cf7d2608be714be9bbc599f270d4b01a P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:27.785398 14081 ts_tablet_manager.cc:1397] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260618 22:39:27.786299 14081 raft_consensus.cc:348] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.786608 14081 raft_consensus.cc:374] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:27.786754 14081 raft_consensus.cc:729] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bdb809d25fe458b98b389dfd4cb2c3b, State: Initialized, Role: FOLLOWER
I20260618 22:39:27.787047 14081 consensus_queue.cc:260] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.791260 14081 ts_tablet_manager.cc:1428] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260618 22:39:27.791965 14079 tablet_bootstrap.cc:492] T cf7d2608be714be9bbc599f270d4b01a P 2c749157c20c458bb3c63cd5b01247b9: No bootstrap required, opened a new log
I20260618 22:39:27.792177 14079 ts_tablet_manager.cc:1397] T cf7d2608be714be9bbc599f270d4b01a P 2c749157c20c458bb3c63cd5b01247b9: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.006s
I20260618 22:39:27.792840 14079 raft_consensus.cc:348] T cf7d2608be714be9bbc599f270d4b01a P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.793045 14079 raft_consensus.cc:374] T cf7d2608be714be9bbc599f270d4b01a P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:27.793128 14079 raft_consensus.cc:729] T cf7d2608be714be9bbc599f270d4b01a P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c749157c20c458bb3c63cd5b01247b9, State: Initialized, Role: FOLLOWER
I20260618 22:39:27.793378 14079 consensus_queue.cc:260] T cf7d2608be714be9bbc599f270d4b01a P 2c749157c20c458bb3c63cd5b01247b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.793821 14080 tablet_bootstrap.cc:492] T cf7d2608be714be9bbc599f270d4b01a P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: No bootstrap required, opened a new log
I20260618 22:39:27.793975 14080 ts_tablet_manager.cc:1397] T cf7d2608be714be9bbc599f270d4b01a P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.004s
I20260618 22:39:27.793992 14079 ts_tablet_manager.cc:1428] T cf7d2608be714be9bbc599f270d4b01a P 2c749157c20c458bb3c63cd5b01247b9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260618 22:39:27.794833 14080 raft_consensus.cc:348] T cf7d2608be714be9bbc599f270d4b01a P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.795125 14080 raft_consensus.cc:374] T cf7d2608be714be9bbc599f270d4b01a P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:27.795228 14080 raft_consensus.cc:729] T cf7d2608be714be9bbc599f270d4b01a P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d7cdbc2b91b4a5ba6f70c4605d8cc23, State: Initialized, Role: FOLLOWER
I20260618 22:39:27.795480 14080 consensus_queue.cc:260] T cf7d2608be714be9bbc599f270d4b01a P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.796190 14080 ts_tablet_manager.cc:1428] T cf7d2608be714be9bbc599f270d4b01a P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260618 22:39:27.850847 14143 raft_consensus.cc:482] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:39:27.851114 14143 raft_consensus.cc:504] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.851974 14143 leader_election.cc:288] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c749157c20c458bb3c63cd5b01247b9 (127.5.213.194:35219), 1d7cdbc2b91b4a5ba6f70c4605d8cc23 (127.5.213.195:37043)
I20260618 22:39:27.852453 13869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7d2608be714be9bbc599f270d4b01a" candidate_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c749157c20c458bb3c63cd5b01247b9" is_pre_election: true
I20260618 22:39:27.852473 14003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7d2608be714be9bbc599f270d4b01a" candidate_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" is_pre_election: true
I20260618 22:39:27.852672 13869 raft_consensus.cc:2454] T cf7d2608be714be9bbc599f270d4b01a P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bdb809d25fe458b98b389dfd4cb2c3b in term 0.
I20260618 22:39:27.852725 14003 raft_consensus.cc:2454] T cf7d2608be714be9bbc599f270d4b01a P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bdb809d25fe458b98b389dfd4cb2c3b in term 0.
I20260618 22:39:27.853156 13658 leader_election.cc:302] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bdb809d25fe458b98b389dfd4cb2c3b, 1d7cdbc2b91b4a5ba6f70c4605d8cc23; no voters:
I20260618 22:39:27.853436 14143 raft_consensus.cc:2781] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:39:27.853547 14143 raft_consensus.cc:482] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:39:27.853636 14143 raft_consensus.cc:3037] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:27.855908 14143 raft_consensus.cc:504] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.856484 14143 leader_election.cc:288] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [CANDIDATE]: Term 1 election: Requested vote from peers 2c749157c20c458bb3c63cd5b01247b9 (127.5.213.194:35219), 1d7cdbc2b91b4a5ba6f70c4605d8cc23 (127.5.213.195:37043)
I20260618 22:39:27.856952 14003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7d2608be714be9bbc599f270d4b01a" candidate_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23"
I20260618 22:39:27.857112 14003 raft_consensus.cc:3037] T cf7d2608be714be9bbc599f270d4b01a P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:27.857084 13869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7d2608be714be9bbc599f270d4b01a" candidate_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c749157c20c458bb3c63cd5b01247b9"
I20260618 22:39:27.857331 13869 raft_consensus.cc:3037] T cf7d2608be714be9bbc599f270d4b01a P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:27.859421 14003 raft_consensus.cc:2454] T cf7d2608be714be9bbc599f270d4b01a P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bdb809d25fe458b98b389dfd4cb2c3b in term 1.
I20260618 22:39:27.859848 13658 leader_election.cc:302] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bdb809d25fe458b98b389dfd4cb2c3b, 1d7cdbc2b91b4a5ba6f70c4605d8cc23; no voters:
I20260618 22:39:27.860095 14143 raft_consensus.cc:2781] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:27.860248 14143 raft_consensus.cc:686] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 LEADER]: Becoming Leader. State: Replica: 1bdb809d25fe458b98b389dfd4cb2c3b, State: Running, Role: LEADER
I20260618 22:39:27.860540 13869 raft_consensus.cc:2454] T cf7d2608be714be9bbc599f270d4b01a P 2c749157c20c458bb3c63cd5b01247b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bdb809d25fe458b98b389dfd4cb2c3b in term 1.
I20260618 22:39:27.860510 14143 consensus_queue.cc:237] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.863266 13566 catalog_manager.cc:5495] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bdb809d25fe458b98b389dfd4cb2c3b (127.5.213.193). New cstate: current_term: 1 leader_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } health_report { overall_health: UNKNOWN } } }
I20260618 22:39:27.899017 14127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:27.894488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43536 (local address 127.5.213.193:40341)
0618 22:39:27.894717 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:27.894722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:27.894742 (+ 20us) server_negotiation.cc:407] Connection header received
0618 22:39:27.894826 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:27.894831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:27.894897 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:27.894970 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:39:27.895321 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.896044 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.897333 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:27.897845 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:27.897952 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:27.898337 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:39:27.898456 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:39:27.898535 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:27.898688 (+ 153us) server_negotiation.cc:299] Negotiation successful
0618 22:39:27.898762 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260618 22:39:27.901626 14003 raft_consensus.cc:1261] T cf7d2608be714be9bbc599f270d4b01a P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 1 FOLLOWER]: Refusing update from remote peer 1bdb809d25fe458b98b389dfd4cb2c3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:39:27.901821 13869 raft_consensus.cc:1261] T cf7d2608be714be9bbc599f270d4b01a P 2c749157c20c458bb3c63cd5b01247b9 [term 1 FOLLOWER]: Refusing update from remote peer 1bdb809d25fe458b98b389dfd4cb2c3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:39:27.902221 14144 consensus_queue.cc:1035] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:27.902942 14144 consensus_queue.cc:1035] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:27.918607 13567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:39:27.918947 13567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:39:27.922062 13567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca: Sending DeleteTablet for 3 replicas of tablet cf7d2608be714be9bbc599f270d4b01a
I20260618 22:39:27.923509 13706 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7d2608be714be9bbc599f270d4b01a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:27 UTC) from {username='oryx', principal='oryx/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:43510
I20260618 22:39:27.923820 14148 tablet_replica.cc:330] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b: stopping tablet replica
I20260618 22:39:27.923817 13845 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7d2608be714be9bbc599f270d4b01a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:27 UTC) from {username='oryx', principal='oryx/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:54286
I20260618 22:39:27.923813 13984 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7d2608be714be9bbc599f270d4b01a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:27 UTC) from {username='oryx', principal='oryx/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:43116
I20260618 22:39:27.924057 14146 tablet_replica.cc:330] T cf7d2608be714be9bbc599f270d4b01a P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: stopping tablet replica
I20260618 22:39:27.924039 14148 raft_consensus.cc:2229] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:39:27.924057 14147 tablet_replica.cc:330] T cf7d2608be714be9bbc599f270d4b01a P 2c749157c20c458bb3c63cd5b01247b9: stopping tablet replica
I20260618 22:39:27.924249 14146 raft_consensus.cc:2229] T cf7d2608be714be9bbc599f270d4b01a P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:39:27.924245 14147 raft_consensus.cc:2229] T cf7d2608be714be9bbc599f270d4b01a P 2c749157c20c458bb3c63cd5b01247b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:39:27.924345 14148 raft_consensus.cc:2258] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:27.924445 14147 raft_consensus.cc:2258] T cf7d2608be714be9bbc599f270d4b01a P 2c749157c20c458bb3c63cd5b01247b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:27.924445 14146 raft_consensus.cc:2258] T cf7d2608be714be9bbc599f270d4b01a P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:27.925414 14146 ts_tablet_manager.cc:1905] T cf7d2608be714be9bbc599f270d4b01a P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:27.925552 14148 ts_tablet_manager.cc:1905] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:27.926101 13569 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:39:27.926733 13569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:39:27.929586 14147 ts_tablet_manager.cc:1905] T cf7d2608be714be9bbc599f270d4b01a P 2c749157c20c458bb3c63cd5b01247b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:27.931814 14146 ts_tablet_manager.cc:1918] T cf7d2608be714be9bbc599f270d4b01a P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:39:27.931955 14146 log.cc:1196] T cf7d2608be714be9bbc599f270d4b01a P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/cf7d2608be714be9bbc599f270d4b01a
I20260618 22:39:27.932323 14146 ts_tablet_manager.cc:1939] T cf7d2608be714be9bbc599f270d4b01a P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Deleting consensus metadata
I20260618 22:39:27.933331 13555 catalog_manager.cc:4841] TS 1d7cdbc2b91b4a5ba6f70c4605d8cc23 (127.5.213.195:37043): tablet cf7d2608be714be9bbc599f270d4b01a (table test-table [id=ae6ab3abefd240ea8119a20da66d299a]) successfully deleted
I20260618 22:39:27.934391 13706 tablet_service.cc:1467] Processing CreateTablet for tablet 8c77158cc4af46a2bc4efea50ae0c9ae (DEFAULT_TABLE table=test-table [id=36aa0b1d7c4f4752b40752c2da59ae7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:27.934918 13706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c77158cc4af46a2bc4efea50ae0c9ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:27.934866 13845 tablet_service.cc:1467] Processing CreateTablet for tablet 8c77158cc4af46a2bc4efea50ae0c9ae (DEFAULT_TABLE table=test-table [id=36aa0b1d7c4f4752b40752c2da59ae7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:27.935279 13845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c77158cc4af46a2bc4efea50ae0c9ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:27.935873 14147 ts_tablet_manager.cc:1918] T cf7d2608be714be9bbc599f270d4b01a P 2c749157c20c458bb3c63cd5b01247b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:39:27.936015 14147 log.cc:1196] T cf7d2608be714be9bbc599f270d4b01a P 2c749157c20c458bb3c63cd5b01247b9: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/cf7d2608be714be9bbc599f270d4b01a
I20260618 22:39:27.936476 14147 ts_tablet_manager.cc:1939] T cf7d2608be714be9bbc599f270d4b01a P 2c749157c20c458bb3c63cd5b01247b9: Deleting consensus metadata
I20260618 22:39:27.938939 13553 catalog_manager.cc:4841] TS 2c749157c20c458bb3c63cd5b01247b9 (127.5.213.194:35219): tablet cf7d2608be714be9bbc599f270d4b01a (table test-table [id=ae6ab3abefd240ea8119a20da66d299a]) successfully deleted
I20260618 22:39:27.939397 13984 tablet_service.cc:1467] Processing CreateTablet for tablet 8c77158cc4af46a2bc4efea50ae0c9ae (DEFAULT_TABLE table=test-table [id=36aa0b1d7c4f4752b40752c2da59ae7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:27.939798 13984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c77158cc4af46a2bc4efea50ae0c9ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:27.941888 14079 tablet_bootstrap.cc:492] T 8c77158cc4af46a2bc4efea50ae0c9ae P 2c749157c20c458bb3c63cd5b01247b9: Bootstrap starting.
I20260618 22:39:27.943547 14079 tablet_bootstrap.cc:654] T 8c77158cc4af46a2bc4efea50ae0c9ae P 2c749157c20c458bb3c63cd5b01247b9: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:27.945581 14148 ts_tablet_manager.cc:1918] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:39:27.945734 14148 log.cc:1196] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/cf7d2608be714be9bbc599f270d4b01a
I20260618 22:39:27.946166 14148 ts_tablet_manager.cc:1939] T cf7d2608be714be9bbc599f270d4b01a P 1bdb809d25fe458b98b389dfd4cb2c3b: Deleting consensus metadata
I20260618 22:39:27.947182 14081 tablet_bootstrap.cc:492] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1bdb809d25fe458b98b389dfd4cb2c3b: Bootstrap starting.
I20260618 22:39:27.948637 14081 tablet_bootstrap.cc:654] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1bdb809d25fe458b98b389dfd4cb2c3b: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:27.948904 13553 catalog_manager.cc:4841] TS 1bdb809d25fe458b98b389dfd4cb2c3b (127.5.213.193:40341): tablet cf7d2608be714be9bbc599f270d4b01a (table test-table [id=ae6ab3abefd240ea8119a20da66d299a]) successfully deleted
I20260618 22:39:27.951753 14081 tablet_bootstrap.cc:492] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1bdb809d25fe458b98b389dfd4cb2c3b: No bootstrap required, opened a new log
I20260618 22:39:27.951895 14081 ts_tablet_manager.cc:1397] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1bdb809d25fe458b98b389dfd4cb2c3b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:39:27.952476 14081 raft_consensus.cc:348] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.952698 14081 raft_consensus.cc:374] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:27.952775 14081 raft_consensus.cc:729] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bdb809d25fe458b98b389dfd4cb2c3b, State: Initialized, Role: FOLLOWER
I20260618 22:39:27.953023 14081 consensus_queue.cc:260] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1bdb809d25fe458b98b389dfd4cb2c3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.954183 14081 ts_tablet_manager.cc:1428] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1bdb809d25fe458b98b389dfd4cb2c3b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260618 22:39:27.954536 14080 tablet_bootstrap.cc:492] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Bootstrap starting.
I20260618 22:39:27.955440 14079 tablet_bootstrap.cc:492] T 8c77158cc4af46a2bc4efea50ae0c9ae P 2c749157c20c458bb3c63cd5b01247b9: No bootstrap required, opened a new log
I20260618 22:39:27.955616 14079 ts_tablet_manager.cc:1397] T 8c77158cc4af46a2bc4efea50ae0c9ae P 2c749157c20c458bb3c63cd5b01247b9: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20260618 22:39:27.956044 14080 tablet_bootstrap.cc:654] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:27.956358 14079 raft_consensus.cc:348] T 8c77158cc4af46a2bc4efea50ae0c9ae P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.956624 14079 raft_consensus.cc:374] T 8c77158cc4af46a2bc4efea50ae0c9ae P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:27.956696 14079 raft_consensus.cc:729] T 8c77158cc4af46a2bc4efea50ae0c9ae P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c749157c20c458bb3c63cd5b01247b9, State: Initialized, Role: FOLLOWER
I20260618 22:39:27.956966 14079 consensus_queue.cc:260] T 8c77158cc4af46a2bc4efea50ae0c9ae P 2c749157c20c458bb3c63cd5b01247b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.959702 14079 ts_tablet_manager.cc:1428] T 8c77158cc4af46a2bc4efea50ae0c9ae P 2c749157c20c458bb3c63cd5b01247b9: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260618 22:39:27.961437 14080 tablet_bootstrap.cc:492] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: No bootstrap required, opened a new log
I20260618 22:39:27.961618 14080 ts_tablet_manager.cc:1397] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260618 22:39:27.962386 14080 raft_consensus.cc:348] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.962734 14080 raft_consensus.cc:374] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:27.962821 14080 raft_consensus.cc:729] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d7cdbc2b91b4a5ba6f70c4605d8cc23, State: Initialized, Role: FOLLOWER
I20260618 22:39:27.963064 14080 consensus_queue.cc:260] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:27.963616 14080 ts_tablet_manager.cc:1428] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260618 22:39:28.225981 14087 raft_consensus.cc:482] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:39:28.226282 14087 raft_consensus.cc:504] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:28.228349 14087 leader_election.cc:288] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bdb809d25fe458b98b389dfd4cb2c3b (127.5.213.193:40341), 2c749157c20c458bb3c63cd5b01247b9 (127.5.213.194:35219)
I20260618 22:39:28.236526 14127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:28.228784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:42655 (local address 127.5.213.193:40341)
0618 22:39:28.228994 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:28.229001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:28.229112 (+ 111us) server_negotiation.cc:407] Connection header received
0618 22:39:28.229396 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:28.229403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:28.229486 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:28.229548 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:28.230239 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.231536 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.234128 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.235400 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.235517 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:28.235589 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:28.236099 (+ 510us) server_negotiation.cc:299] Negotiation successful
0618 22:39:28.236229 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260618 22:39:28.237326 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:28.228500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:40341 (local address 127.5.213.195:42655)
0618 22:39:28.229017 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:28.229036 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:28.229218 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:28.229742 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:28.229747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:28.229784 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:28.230058 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:28.230069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.231751 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.231757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:28.233932 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:28.233950 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.235747 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.235753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:28.235935 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:28.235967 (+ 32us) client_negotiation.cc:769] Sending connection context
0618 22:39:28.236892 (+ 925us) client_negotiation.cc:240] Negotiation successful
0618 22:39:28.237029 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":226,"threads_started":1}
I20260618 22:39:28.245018 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:28.229102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:35219 (local address 127.5.213.195:36759)
0618 22:39:28.237697 (+ 8595us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:28.237715 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:28.237823 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:28.238640 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:28.238646 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:28.238672 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:28.238831 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:28.238841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.240425 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.240431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:28.242085 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:28.242098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.244306 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.244312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:28.244504 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:28.244540 (+ 36us) client_negotiation.cc:769] Sending connection context
0618 22:39:28.244596 (+ 56us) client_negotiation.cc:240] Negotiation successful
0618 22:39:28.244727 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8499,"threads_started":1}
I20260618 22:39:28.245980 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:28.229314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:36759 (local address 127.5.213.194:35219)
0618 22:39:28.238334 (+ 9020us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:28.238340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:28.238359 (+ 19us) server_negotiation.cc:407] Connection header received
0618 22:39:28.238415 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:28.238420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:28.238492 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:28.238958 (+ 466us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:28.239106 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.240281 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.242548 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.244123 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.245398 (+ 1275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:28.245468 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:28.245512 (+ 44us) server_negotiation.cc:299] Negotiation successful
0618 22:39:28.245638 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8878,"thread_start_us":8704,"threads_started":1}
I20260618 22:39:28.247243 13869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c77158cc4af46a2bc4efea50ae0c9ae" candidate_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c749157c20c458bb3c63cd5b01247b9" is_pre_election: true
I20260618 22:39:28.247292 13726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c77158cc4af46a2bc4efea50ae0c9ae" candidate_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" is_pre_election: true
I20260618 22:39:28.247493 13869 raft_consensus.cc:2454] T 8c77158cc4af46a2bc4efea50ae0c9ae P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d7cdbc2b91b4a5ba6f70c4605d8cc23 in term 0.
I20260618 22:39:28.247608 13726 raft_consensus.cc:2454] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d7cdbc2b91b4a5ba6f70c4605d8cc23 in term 0.
I20260618 22:39:28.248152 13935 leader_election.cc:302] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d7cdbc2b91b4a5ba6f70c4605d8cc23, 2c749157c20c458bb3c63cd5b01247b9; no voters:
I20260618 22:39:28.248581 14087 raft_consensus.cc:2781] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:39:28.248723 14087 raft_consensus.cc:482] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:39:28.248822 14087 raft_consensus.cc:3037] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:28.251991 14087 raft_consensus.cc:504] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:28.252760 14087 leader_election.cc:288] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [CANDIDATE]: Term 1 election: Requested vote from peers 1bdb809d25fe458b98b389dfd4cb2c3b (127.5.213.193:40341), 2c749157c20c458bb3c63cd5b01247b9 (127.5.213.194:35219)
I20260618 22:39:28.253223 13726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c77158cc4af46a2bc4efea50ae0c9ae" candidate_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b"
I20260618 22:39:28.253454 13726 raft_consensus.cc:3037] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1bdb809d25fe458b98b389dfd4cb2c3b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:28.253430 13869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c77158cc4af46a2bc4efea50ae0c9ae" candidate_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c749157c20c458bb3c63cd5b01247b9"
I20260618 22:39:28.253646 13869 raft_consensus.cc:3037] T 8c77158cc4af46a2bc4efea50ae0c9ae P 2c749157c20c458bb3c63cd5b01247b9 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:28.256819 13869 raft_consensus.cc:2454] T 8c77158cc4af46a2bc4efea50ae0c9ae P 2c749157c20c458bb3c63cd5b01247b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d7cdbc2b91b4a5ba6f70c4605d8cc23 in term 1.
I20260618 22:39:28.256825 13726 raft_consensus.cc:2454] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d7cdbc2b91b4a5ba6f70c4605d8cc23 in term 1.
I20260618 22:39:28.257648 13935 leader_election.cc:302] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d7cdbc2b91b4a5ba6f70c4605d8cc23, 2c749157c20c458bb3c63cd5b01247b9; no voters:
I20260618 22:39:28.257943 14087 raft_consensus.cc:2781] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:28.258432 14087 raft_consensus.cc:686] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 1 LEADER]: Becoming Leader. State: Replica: 1d7cdbc2b91b4a5ba6f70c4605d8cc23, State: Running, Role: LEADER
I20260618 22:39:28.258919 14087 consensus_queue.cc:237] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } }
I20260618 22:39:28.262933 13569 catalog_manager.cc:5495] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d7cdbc2b91b4a5ba6f70c4605d8cc23 (127.5.213.195). New cstate: current_term: 1 leader_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d7cdbc2b91b4a5ba6f70c4605d8cc23" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 37043 } health_report { overall_health: HEALTHY } } }
I20260618 22:39:28.332548 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:28.325377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43126 (local address 127.5.213.195:37043)
0618 22:39:28.326056 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:28.326062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:28.326089 (+ 27us) server_negotiation.cc:407] Connection header received
0618 22:39:28.326198 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:28.326204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:28.326329 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:28.326547 (+ 218us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:39:28.326960 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.328186 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.329616 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.330279 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.330423 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:28.330871 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:39:28.331850 (+ 979us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:39:28.332011 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:28.332091 (+ 80us) server_negotiation.cc:299] Negotiation successful
0618 22:39:28.332174 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":409,"threads_started":1}
I20260618 22:39:28.348222 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:28.342315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:40341 (local address 127.5.213.195:46085)
0618 22:39:28.342977 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:28.342992 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:28.343099 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:28.343448 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:28.343452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:28.343460 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:39:28.343752 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:28.343760 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.345182 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.345186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:28.346211 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:28.346223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.347233 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.347237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:28.347341 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:28.347392 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:39:28.347786 (+ 394us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:39:28.347798 (+ 12us) client_negotiation.cc:769] Sending connection context
0618 22:39:28.347849 (+ 51us) client_negotiation.cc:240] Negotiation successful
0618 22:39:28.347942 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":233,"threads_started":1}
I20260618 22:39:28.348250 14127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:28.342555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:46085 (local address 127.5.213.193:40341)
0618 22:39:28.342732 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:28.342737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:28.343088 (+ 351us) server_negotiation.cc:407] Connection header received
0618 22:39:28.343228 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:28.343234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:28.343343 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:28.343440 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:39:28.343919 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.345048 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.346346 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.347081 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.347212 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:28.347567 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:39:28.347699 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:39:28.347788 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:28.347894 (+ 106us) server_negotiation.cc:299] Negotiation successful
0618 22:39:28.347973 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 22:39:28.356379 13869 raft_consensus.cc:1261] T 8c77158cc4af46a2bc4efea50ae0c9ae P 2c749157c20c458bb3c63cd5b01247b9 [term 1 FOLLOWER]: Refusing update from remote peer 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:39:28.356483 13726 raft_consensus.cc:1261] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 FOLLOWER]: Refusing update from remote peer 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:39:28.357084 14174 consensus_queue.cc:1035] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c749157c20c458bb3c63cd5b01247b9" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:28.357446 14087 consensus_queue.cc:1035] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bdb809d25fe458b98b389dfd4cb2c3b" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 40341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:28.387866 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:28.380967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:60405 (local address 127.5.213.195:37043)
0618 22:39:28.381120 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:28.381126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:28.383201 (+ 2075us) server_negotiation.cc:407] Connection header received
0618 22:39:28.383365 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:28.383370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:28.383432 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:28.383500 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:39:28.384026 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.384708 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.386301 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.386744 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.386852 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:28.387234 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:39:28.387334 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:39:28.387403 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:28.387529 (+ 126us) server_negotiation.cc:299] Negotiation successful
0618 22:39:28.387603 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:39:28.388242 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:28.380919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:37043 (local address 127.5.213.193:60405)
0618 22:39:28.383108 (+ 2189us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:28.383124 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:28.383245 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:28.383531 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:28.383535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:28.383547 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:39:28.383880 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:28.383889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.384826 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.384832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:28.386170 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:28.386182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.386911 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.386918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:28.387057 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:28.387095 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:39:28.387424 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:39:28.387435 (+ 11us) client_negotiation.cc:769] Sending connection context
0618 22:39:28.387503 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 22:39:28.387631 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2047,"thread_start_us":1892,"threads_started":1}
I20260618 22:39:28.422860 13569 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:39:28.423098 13569 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:39:28.426553 13569 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3284b18e08334919b3a7fd0c05a13aca: Sending DeleteTablet for 3 replicas of tablet 8c77158cc4af46a2bc4efea50ae0c9ae
I20260618 22:39:28.427357 13706 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c77158cc4af46a2bc4efea50ae0c9ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:28 UTC) from {username='oryx', principal='oryx/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:43510
I20260618 22:39:28.427549 13845 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c77158cc4af46a2bc4efea50ae0c9ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:28 UTC) from {username='oryx', principal='oryx/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:54286
I20260618 22:39:28.427645 14148 tablet_replica.cc:330] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1bdb809d25fe458b98b389dfd4cb2c3b: stopping tablet replica
I20260618 22:39:28.427819 14148 raft_consensus.cc:2229] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:39:28.428062 14148 raft_consensus.cc:2258] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1bdb809d25fe458b98b389dfd4cb2c3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:28.428200 13984 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c77158cc4af46a2bc4efea50ae0c9ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:28 UTC) from {username='oryx', principal='oryx/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:43116
I20260618 22:39:28.428488 14146 tablet_replica.cc:330] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: stopping tablet replica
I20260618 22:39:28.428658 14146 raft_consensus.cc:2229] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:39:28.428679 14147 tablet_replica.cc:330] T 8c77158cc4af46a2bc4efea50ae0c9ae P 2c749157c20c458bb3c63cd5b01247b9: stopping tablet replica
I20260618 22:39:28.428848 14147 raft_consensus.cc:2229] T 8c77158cc4af46a2bc4efea50ae0c9ae P 2c749157c20c458bb3c63cd5b01247b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:39:28.429016 14146 raft_consensus.cc:2258] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:28.429019 14148 ts_tablet_manager.cc:1905] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1bdb809d25fe458b98b389dfd4cb2c3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:28.429020 14147 raft_consensus.cc:2258] T 8c77158cc4af46a2bc4efea50ae0c9ae P 2c749157c20c458bb3c63cd5b01247b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:28.429891 14146 ts_tablet_manager.cc:1905] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:28.430294 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 13640
I20260618 22:39:28.430465 14147 ts_tablet_manager.cc:1905] T 8c77158cc4af46a2bc4efea50ae0c9ae P 2c749157c20c458bb3c63cd5b01247b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:28.435447 14148 ts_tablet_manager.cc:1918] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1bdb809d25fe458b98b389dfd4cb2c3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:39:28.435642 14148 log.cc:1196] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1bdb809d25fe458b98b389dfd4cb2c3b: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/8c77158cc4af46a2bc4efea50ae0c9ae
I20260618 22:39:28.436093 14148 ts_tablet_manager.cc:1939] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1bdb809d25fe458b98b389dfd4cb2c3b: Deleting consensus metadata
I20260618 22:39:28.437059 14146 ts_tablet_manager.cc:1918] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:39:28.437186 14146 log.cc:1196] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/8c77158cc4af46a2bc4efea50ae0c9ae
I20260618 22:39:28.437158 13553 catalog_manager.cc:4841] TS 1bdb809d25fe458b98b389dfd4cb2c3b (127.5.213.193:40341): tablet 8c77158cc4af46a2bc4efea50ae0c9ae (table test-table [id=36aa0b1d7c4f4752b40752c2da59ae7b]) successfully deleted
I20260618 22:39:28.437575 14146 ts_tablet_manager.cc:1939] T 8c77158cc4af46a2bc4efea50ae0c9ae P 1d7cdbc2b91b4a5ba6f70c4605d8cc23: Deleting consensus metadata
I20260618 22:39:28.437927 14147 ts_tablet_manager.cc:1918] T 8c77158cc4af46a2bc4efea50ae0c9ae P 2c749157c20c458bb3c63cd5b01247b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:39:28.438055 14147 log.cc:1196] T 8c77158cc4af46a2bc4efea50ae0c9ae P 2c749157c20c458bb3c63cd5b01247b9: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/8c77158cc4af46a2bc4efea50ae0c9ae
I20260618 22:39:28.438402 14147 ts_tablet_manager.cc:1939] T 8c77158cc4af46a2bc4efea50ae0c9ae P 2c749157c20c458bb3c63cd5b01247b9: Deleting consensus metadata
I20260618 22:39:28.438511 13555 catalog_manager.cc:4841] TS 1d7cdbc2b91b4a5ba6f70c4605d8cc23 (127.5.213.195:37043): tablet 8c77158cc4af46a2bc4efea50ae0c9ae (table test-table [id=36aa0b1d7c4f4752b40752c2da59ae7b]) successfully deleted
I20260618 22:39:28.439302 13553 catalog_manager.cc:4841] TS 2c749157c20c458bb3c63cd5b01247b9 (127.5.213.194:35219): tablet 8c77158cc4af46a2bc4efea50ae0c9ae (table test-table [id=36aa0b1d7c4f4752b40752c2da59ae7b]) successfully deleted
Jun 18 22:39:28 dist-test-slave-lnj3 krb5kdc[13524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.213.193@KRBTEST.COM
I20260618 22:39:28.447429 14127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:28.431479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43544 (local address 127.5.213.193:40341)
0618 22:39:28.431613 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:28.431619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:28.431697 (+ 78us) server_negotiation.cc:407] Connection header received
0618 22:39:28.431925 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:28.431931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:28.432001 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:28.432062 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:28.433694 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.434848 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.436726 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.437335 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.437428 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:28.441975 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:28.441987 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:28.442002 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:28.442038 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:28.444758 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:28.445638 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:28.445645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:28.445647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:28.445777 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:28.446146 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:28.446149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:28.446151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:28.446488 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:28.446791 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:28.447040 (+ 249us) server_negotiation.cc:299] Negotiation successful
0618 22:39:28.447196 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:39:28.448256 13771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:28.587669 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 13640
I20260618 22:39:28.609097 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 13780
Jun 18 22:39:28 dist-test-slave-lnj3 krb5kdc[13524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.213.194@KRBTEST.COM
I20260618 22:39:28.624791 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:28.609873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54314 (local address 127.5.213.194:35219)
0618 22:39:28.610034 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:28.610038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:28.610090 (+ 52us) server_negotiation.cc:407] Connection header received
0618 22:39:28.610249 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:28.610253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:28.610319 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:28.610397 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:28.611828 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.612521 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.614210 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.614616 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.614948 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:28.619479 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:28.619491 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:28.619505 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:28.619537 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:28.622126 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:28.622851 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:28.622859 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:28.622862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:28.622988 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:28.623337 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:28.623340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:28.623342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:28.623684 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:28.623960 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:28.624271 (+ 311us) server_negotiation.cc:299] Negotiation successful
0618 22:39:28.624505 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260618 22:39:28.625358 13910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:28.758128 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 13780
I20260618 22:39:28.784276 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 13919
Jun 18 22:39:28 dist-test-slave-lnj3 krb5kdc[13524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.213.195@KRBTEST.COM
I20260618 22:39:28.800156 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:28.785032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.5.213.195:37043)
0618 22:39:28.785228 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:28.785234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:28.785256 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:39:28.785384 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:28.785389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:28.785461 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:28.785570 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:28.786742 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.787813 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.789081 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:28.789801 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:28.789919 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:28.794214 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:28.794225 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:28.794242 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:28.794282 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:28.797299 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:28.798076 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:28.798083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:28.798086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:28.798281 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:28.798736 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:28.798740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:28.798743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:28.799103 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:28.799330 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:28.799643 (+ 313us) server_negotiation.cc:299] Negotiation successful
0618 22:39:28.799800 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260618 22:39:28.800817 14048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:28.930454 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 13919
I20260618 22:39:28.947763 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 13537
I20260618 22:39:28.948837 13609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:29.060397 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 13537
2026-06-18T22:39:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6576 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:39:29 dist-test-slave-lnj3 krb5kdc[14196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:39:29 dist-test-slave-lnj3 krb5kdc[14196](info): set up 2 sockets
Jun 18 22:39:29 dist-test-slave-lnj3 krb5kdc[14196](info): commencing operation
krb5kdc: starting...
W20260618 22:39:30.075325 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:39:30 dist-test-slave-lnj3 krb5kdc[14196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:39:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:39:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:39:30.271636 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:45579
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:43693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:45579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:30.580267 14209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:30.580590 14209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:30.580731 14209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:30.588213 14209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:39:30.588338 14209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:30.588402 14209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:30.588478 14209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:39:30.588536 14209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:39:30.598546 14209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:43693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:45579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:45579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:30.599895 14209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:30.601203 14209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:30.612596 14214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:30.612622 14215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:30.612728 14217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:30.613276 14209 server_base.cc:1034] running on GCE node
I20260618 22:39:30.613997 14209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:30.615424 14209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:30.616614 14209 hybrid_clock.cc:648] HybridClock initialized: now 1781822370616565 us; error 45 us; skew 500 ppm
I20260618 22:39:30.616942 14209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-18T22:39:30Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1620 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:39:30 dist-test-slave-lnj3 krb5kdc[14223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:39:30 dist-test-slave-lnj3 krb5kdc[14223](info): set up 2 sockets
Jun 18 22:39:30 dist-test-slave-lnj3 krb5kdc[14223](info): commencing operation
krb5kdc: starting...
W20260618 22:39:31.695209 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.006s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:39:31 dist-test-slave-lnj3 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:39:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:39:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:39:31.889993 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:43463
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:43463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:32.209424 14236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:32.209748 14236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:32.209888 14236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:32.217175 14236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:39:32.217316 14236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:32.217404 14236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:32.217485 14236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:39:32.217558 14236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:39:32.227718 14236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:43463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:43463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:32.229066 14236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:32.230396 14236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:32.242264 14244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:32.242250 14241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:32.242247 14242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:32.243149 14236 server_base.cc:1034] running on GCE node
I20260618 22:39:32.243934 14236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:32.245303 14236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:32.246503 14236 hybrid_clock.cc:648] HybridClock initialized: now 1781822372246443 us; error 54 us; skew 500 ppm
I20260618 22:39:32.246860 14236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:32 dist-test-slave-lnj3 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822372, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:32.249740 14236 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:39:32.251181 14236 webserver.cc:460] Webserver started at http://127.5.213.254:39677/ using document root <none> and password file <none>
I20260618 22:39:32.251812 14236 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:32.251905 14236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:32.252146 14236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:32.254976 14236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "2dd472574dde4b2f9168813488669463"
format_stamp: "Formatted at 2026-06-18 22:39:32 on dist-test-slave-lnj3"
server_key: "70632e054d6ad306a5cfd35825766733"
server_key_iv: "c94be0f31b5b8e5377359fb2f498d478"
server_key_version: "encryptionkey@0"
I20260618 22:39:32.255671 14236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "2dd472574dde4b2f9168813488669463"
format_stamp: "Formatted at 2026-06-18 22:39:32 on dist-test-slave-lnj3"
server_key: "70632e054d6ad306a5cfd35825766733"
server_key_iv: "c94be0f31b5b8e5377359fb2f498d478"
server_key_version: "encryptionkey@0"
I20260618 22:39:32.260528 14236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:39:32.264039 14251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:32.265377 14236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:39:32.265520 14236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "2dd472574dde4b2f9168813488669463"
format_stamp: "Formatted at 2026-06-18 22:39:32 on dist-test-slave-lnj3"
server_key: "70632e054d6ad306a5cfd35825766733"
server_key_iv: "c94be0f31b5b8e5377359fb2f498d478"
server_key_version: "encryptionkey@0"
I20260618 22:39:32.265697 14236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:32.284251 14236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:32.288527 14236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:32.288781 14236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:32.304976 14236 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:43463
I20260618 22:39:32.304991 14302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:43463 every 8 connection(s)
I20260618 22:39:32.306290 14236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:39:32.308318 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 14236
I20260618 22:39:32.308498 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:39:32.309094 5975 external_mini_cluster.cc:1428] Setting key 5a49042f6740f92c8fe5f9720f5c4d19
I20260618 22:39:32.310019 14303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:32.321439 14303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463: Bootstrap starting.
Jun 18 22:39:32 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:39:32.324788 14303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:32.325834 14303 log.cc:824] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:32.328444 14303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463: No bootstrap required, opened a new log
I20260618 22:39:32.329622 14306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:32.310914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57104 (local address 127.5.213.254:43463)
0618 22:39:32.311616 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:32.311627 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:32.311667 (+ 40us) server_negotiation.cc:407] Connection header received
0618 22:39:32.313226 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:32.313247 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:32.313645 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:32.314027 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:32.315496 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:32.316787 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:32.317952 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:32.318698 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:32.318869 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:32.322486 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:32.322494 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:32.322508 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:32.322542 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:32.325694 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:32.326447 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:32.326457 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:32.326465 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:32.326686 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:32.327031 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:32.327036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:32.327040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:32.327546 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:32.327803 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:32.328241 (+ 438us) server_negotiation.cc:299] Negotiation successful
0618 22:39:32.328623 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":311,"threads_started":1}
I20260618 22:39:32.336184 14303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd472574dde4b2f9168813488669463" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43463 } }
I20260618 22:39:32.336536 14303 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:32.336621 14303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd472574dde4b2f9168813488669463, State: Initialized, Role: FOLLOWER
I20260618 22:39:32.337250 14303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd472574dde4b2f9168813488669463" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43463 } }
I20260618 22:39:32.337478 14303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:39:32.337617 14303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:39:32.337787 14303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:32.339838 14303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd472574dde4b2f9168813488669463" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43463 } }
I20260618 22:39:32.340308 14303 leader_election.cc:302] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dd472574dde4b2f9168813488669463; no voters:
I20260618 22:39:32.340848 14303 leader_election.cc:288] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:39:32.340965 14308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:32.341557 14308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [term 1 LEADER]: Becoming Leader. State: Replica: 2dd472574dde4b2f9168813488669463, State: Running, Role: LEADER
I20260618 22:39:32.342072 14308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd472574dde4b2f9168813488669463" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43463 } }
I20260618 22:39:32.342717 14303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:39:32.344558 14309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dd472574dde4b2f9168813488669463" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd472574dde4b2f9168813488669463" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43463 } } }
I20260618 22:39:32.344858 14309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:32.345616 14313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:39:32.345502 14310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dd472574dde4b2f9168813488669463. Latest consensus state: current_term: 1 leader_uuid: "2dd472574dde4b2f9168813488669463" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd472574dde4b2f9168813488669463" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43463 } } }
I20260618 22:39:32.346648 14310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:32.350788 14313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:39:32.360749 14313 catalog_manager.cc:1347] Generated new cluster ID: f04b1f5ea0c34a0c90e6702b33ed7471
I20260618 22:39:32.360862 14313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:39:32.383553 14313 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:39:32.384852 14313 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:39:32.395742 14313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463: Generated new TSK 0
I20260618 22:39:32.396628 14313 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
I20260618 22:39:32.452488 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:43463
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:32.801177 14329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:32.801465 14329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:32.801586 14329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:32.809095 14329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:32.809234 14329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:32.809381 14329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:39:32.819446 14329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:43463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:32.820806 14329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:32.822109 14329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:32.837657 14334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:32.837657 14335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:32.837699 14337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:32.838505 14329 server_base.cc:1034] running on GCE node
I20260618 22:39:32.839159 14329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:32.840077 14329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:32.841291 14329 hybrid_clock.cc:648] HybridClock initialized: now 1781822372841240 us; error 45 us; skew 500 ppm
I20260618 22:39:32.841599 14329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:32 dist-test-slave-lnj3 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822372, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:32.845072 14329 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:39:32.846573 14329 webserver.cc:460] Webserver started at http://127.5.213.193:43163/ using document root <none> and password file <none>
I20260618 22:39:32.847317 14329 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:32.847415 14329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:32.847671 14329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:32.850570 14329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "4e8745d01e2c4fb98ba3323b05dbe9ec"
format_stamp: "Formatted at 2026-06-18 22:39:32 on dist-test-slave-lnj3"
server_key: "174d6506b397abc31bba3087872c3584"
server_key_iv: "efd01f50a12a3ae20e9b26efcc073d18"
server_key_version: "encryptionkey@0"
I20260618 22:39:32.851383 14329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "4e8745d01e2c4fb98ba3323b05dbe9ec"
format_stamp: "Formatted at 2026-06-18 22:39:32 on dist-test-slave-lnj3"
server_key: "174d6506b397abc31bba3087872c3584"
server_key_iv: "efd01f50a12a3ae20e9b26efcc073d18"
server_key_version: "encryptionkey@0"
I20260618 22:39:32.856679 14329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:39:32.860213 14344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:32.861586 14329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260618 22:39:32.861730 14329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "4e8745d01e2c4fb98ba3323b05dbe9ec"
format_stamp: "Formatted at 2026-06-18 22:39:32 on dist-test-slave-lnj3"
server_key: "174d6506b397abc31bba3087872c3584"
server_key_iv: "efd01f50a12a3ae20e9b26efcc073d18"
server_key_version: "encryptionkey@0"
I20260618 22:39:32.861919 14329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:32.873488 14329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:32.877965 14329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:32.878227 14329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:32.879225 14329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:32.880548 14329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:32.880618 14329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:32.880722 14329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:32.880777 14329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:32.910811 14329 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:43995
I20260618 22:39:32.910845 14456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:43995 every 8 connection(s)
I20260618 22:39:32.912042 14329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:39:32.915964 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 14329
I20260618 22:39:32.916160 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:39:32.916800 5975 external_mini_cluster.cc:1428] Setting key 3d674f2c99bd81e931901aadad061fae
Jun 18 22:39:32 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822372, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:39:32.931984 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:32.914516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:38521 (local address 127.5.213.254:43463)
0618 22:39:32.914965 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:32.914970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:32.915923 (+ 953us) server_negotiation.cc:407] Connection header received
0618 22:39:32.917868 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:32.917872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:32.917930 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:32.918021 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:32.919712 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:32.920800 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:32.922373 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:32.922950 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:32.923069 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:32.926515 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:32.926521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:32.926524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:32.926555 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:32.928908 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:32.929722 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:32.929726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:32.929728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:32.929845 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:32.930298 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:32.930302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:32.930304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:32.930571 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:32.930767 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:32.931537 (+ 770us) server_negotiation.cc:299] Negotiation successful
0618 22:39:32.931734 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":349,"thread_start_us":222,"threads_started":1}
I20260618 22:39:32.933390 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:32.915090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.193:38521)
0618 22:39:32.915742 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:32.915805 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:32.917619 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:32.918283 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:32.918294 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:32.918993 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:32.919549 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:32.919566 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:32.920940 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:32.920944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:32.922241 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:32.922253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:32.923134 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:32.923139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:32.923337 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:32.924012 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:32.924034 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:32.926290 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:32.929044 (+ 2754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:32.929053 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:32.929071 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:32.929587 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:32.929941 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:32.929946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:32.929949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:32.930161 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:32.930757 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:32.930766 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:32.931176 (+ 410us) client_negotiation.cc:769] Sending connection context
0618 22:39:32.931458 (+ 282us) client_negotiation.cc:240] Negotiation successful
0618 22:39:32.931991 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":236,"threads_started":1}
I20260618 22:39:32.935163 14457 heartbeater.cc:347] Connected to a master server at 127.5.213.254:43463
I20260618 22:39:32.935490 14457 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:32.936190 14457 heartbeater.cc:511] Master 127.5.213.254:43463 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.213.194@KRBTEST.COM; defaulting to no policy
I20260618 22:39:32.938428 14268 ts_manager.cc:194] Registered new tserver with Master: 4e8745d01e2c4fb98ba3323b05dbe9ec (127.5.213.193:43995)
I20260618 22:39:32.939872 14268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:38521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
I20260618 22:39:32.963348 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:43463
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:33.270705 14463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:33.271000 14463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:33.271139 14463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:33.278417 14463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:33.278548 14463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:33.278724 14463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:39:33.288365 14463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:43463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:33.289674 14463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:33.290978 14463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:33.303599 14468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:33.303627 14469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:33.303839 14471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:33.304571 14463 server_base.cc:1034] running on GCE node
I20260618 22:39:33.305173 14463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:33.306077 14463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:33.307272 14463 hybrid_clock.cc:648] HybridClock initialized: now 1781822373307213 us; error 49 us; skew 500 ppm
I20260618 22:39:33.307561 14463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:33 dist-test-slave-lnj3 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822373, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:33.310596 14463 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:39:33.319375 14463 webserver.cc:460] Webserver started at http://127.5.213.194:42201/ using document root <none> and password file <none>
I20260618 22:39:33.320051 14463 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:33.320173 14463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:33.320449 14463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:33.323400 14463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "bb06623180d74f53bc40f82927ce8c76"
format_stamp: "Formatted at 2026-06-18 22:39:33 on dist-test-slave-lnj3"
server_key: "c260f0cf37dbddef32ce21c95594863a"
server_key_iv: "20dd768744caa8a484999bb35ba65ea2"
server_key_version: "encryptionkey@0"
I20260618 22:39:33.324141 14463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "bb06623180d74f53bc40f82927ce8c76"
format_stamp: "Formatted at 2026-06-18 22:39:33 on dist-test-slave-lnj3"
server_key: "c260f0cf37dbddef32ce21c95594863a"
server_key_iv: "20dd768744caa8a484999bb35ba65ea2"
server_key_version: "encryptionkey@0"
I20260618 22:39:33.329406 14463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260618 22:39:33.333053 14478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:33.334537 14463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260618 22:39:33.334738 14463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "bb06623180d74f53bc40f82927ce8c76"
format_stamp: "Formatted at 2026-06-18 22:39:33 on dist-test-slave-lnj3"
server_key: "c260f0cf37dbddef32ce21c95594863a"
server_key_iv: "20dd768744caa8a484999bb35ba65ea2"
server_key_version: "encryptionkey@0"
I20260618 22:39:33.334940 14463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:33.358974 14463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:33.362891 14463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:33.363142 14463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:33.364102 14463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:33.365381 14463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:33.365453 14463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:33.365564 14463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:33.365612 14463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:33.393736 14463 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:41549
I20260618 22:39:33.393779 14590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:41549 every 8 connection(s)
I20260618 22:39:33.394985 14463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:39:33.402611 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 14463
I20260618 22:39:33.402813 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:39:33.403314 5975 external_mini_cluster.cc:1428] Setting key e84adae51df1f7c518e40be37fbeac10
Jun 18 22:39:33 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822373, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:39:33.414474 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:33.397355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:40257 (local address 127.5.213.254:43463)
0618 22:39:33.397524 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:33.397528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:33.398701 (+ 1173us) server_negotiation.cc:407] Connection header received
0618 22:39:33.400483 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:33.400488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:33.400565 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:33.400649 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:33.402130 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:33.403166 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:33.404891 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:33.405421 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:33.405539 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:33.408823 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:33.408828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:33.408833 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:33.408866 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:33.411708 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:33.412492 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:33.412495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:33.412498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:33.412608 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:33.413012 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:33.413015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:33.413017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:33.413241 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:33.413344 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:33.414091 (+ 747us) server_negotiation.cc:299] Negotiation successful
0618 22:39:33.414249 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260618 22:39:33.415854 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:33.397867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.194:40257)
0618 22:39:33.398532 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:33.398572 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:33.400215 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:33.400882 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:33.400890 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:33.401426 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:33.401965 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:33.401982 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:33.403310 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:33.403314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:33.404758 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:33.404770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:33.405624 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:33.405629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:33.405823 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:33.406447 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:33.406466 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:33.408608 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:33.411836 (+ 3228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:33.411845 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:33.411862 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:33.412397 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:33.412701 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:33.412706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:33.412710 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:33.412933 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:33.413351 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:33.413358 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:33.413731 (+ 373us) client_negotiation.cc:769] Sending connection context
0618 22:39:33.414013 (+ 282us) client_negotiation.cc:240] Negotiation successful
0618 22:39:33.414474 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":243,"threads_started":1}
I20260618 22:39:33.417176 14591 heartbeater.cc:347] Connected to a master server at 127.5.213.254:43463
I20260618 22:39:33.417472 14591 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:33.418130 14591 heartbeater.cc:511] Master 127.5.213.254:43463 requested a full tablet report, sending...
I20260618 22:39:33.419489 14268 ts_manager.cc:194] Registered new tserver with Master: bb06623180d74f53bc40f82927ce8c76 (127.5.213.194:41549)
I20260618 22:39:33.420288 14268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:40257
WARNING: no policy specified for kudu/127.5.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
I20260618 22:39:33.447613 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:43463
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:33.753083 14596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:33.753401 14596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:33.753561 14596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:33.760807 14596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:33.760932 14596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:33.761090 14596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:39:33.770814 14596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:43463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:33.772004 14596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:33.773247 14596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:33.785216 14601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:33.785136 14604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:33.785245 14602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:33.785743 14596 server_base.cc:1034] running on GCE node
I20260618 22:39:33.786305 14596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:33.787156 14596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:33.788340 14596 hybrid_clock.cc:648] HybridClock initialized: now 1781822373788300 us; error 33 us; skew 500 ppm
I20260618 22:39:33.788628 14596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:33 dist-test-slave-lnj3 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822373, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:33.791546 14596 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:39:33.792977 14596 webserver.cc:460] Webserver started at http://127.5.213.195:39463/ using document root <none> and password file <none>
I20260618 22:39:33.793607 14596 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:33.793730 14596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:33.793996 14596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:33.796835 14596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "bfc429e81766413dba69afbfc6fa1871"
format_stamp: "Formatted at 2026-06-18 22:39:33 on dist-test-slave-lnj3"
server_key: "3b385f05e1fcf5e027a5eb1d1205715a"
server_key_iv: "e34f965bbc60d1754b57c6cc38b3cf4e"
server_key_version: "encryptionkey@0"
I20260618 22:39:33.797554 14596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "bfc429e81766413dba69afbfc6fa1871"
format_stamp: "Formatted at 2026-06-18 22:39:33 on dist-test-slave-lnj3"
server_key: "3b385f05e1fcf5e027a5eb1d1205715a"
server_key_iv: "e34f965bbc60d1754b57c6cc38b3cf4e"
server_key_version: "encryptionkey@0"
I20260618 22:39:33.802321 14596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260618 22:39:33.805622 14611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:33.806951 14596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:39:33.807089 14596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "bfc429e81766413dba69afbfc6fa1871"
format_stamp: "Formatted at 2026-06-18 22:39:33 on dist-test-slave-lnj3"
server_key: "3b385f05e1fcf5e027a5eb1d1205715a"
server_key_iv: "e34f965bbc60d1754b57c6cc38b3cf4e"
server_key_version: "encryptionkey@0"
I20260618 22:39:33.807260 14596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:33.835625 14596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:33.839591 14596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:33.839834 14596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:33.840699 14596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:33.841910 14596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:33.841984 14596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:33.842096 14596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:33.842146 14596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:33.871685 14596 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:43741
I20260618 22:39:33.871763 14723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:43741 every 8 connection(s)
I20260618 22:39:33.872923 14596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:39:33.874239 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 14596
I20260618 22:39:33.874385 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:39:33.874895 5975 external_mini_cluster.cc:1428] Setting key 1112752fcbd6dfca0d8fc137382f5b70
Jun 18 22:39:33 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822373, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:39:33.892166 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:33.875404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:44129 (local address 127.5.213.254:43463)
0618 22:39:33.875597 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:33.875603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:33.876949 (+ 1346us) server_negotiation.cc:407] Connection header received
0618 22:39:33.878842 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:33.878847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:33.878924 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:33.879008 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:33.880413 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:33.881142 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:33.883053 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:33.883465 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:33.883571 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:33.886737 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:33.886742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:33.886745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:33.886775 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:33.889309 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:33.890043 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:33.890047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:33.890050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:33.890176 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:33.890550 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:33.890555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:33.890559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:33.890852 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:33.890976 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:33.891629 (+ 653us) server_negotiation.cc:299] Negotiation successful
0618 22:39:33.891842 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 22:39:33.893074 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:33.875911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.195:44129)
0618 22:39:33.876814 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:33.876856 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:33.878577 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:33.879248 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:33.879255 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:33.879730 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:33.880263 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:33.880280 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:33.881269 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:33.881274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:33.882931 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:33.882943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:33.883664 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:33.883671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:33.883890 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:33.884426 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:33.884446 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:33.886478 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:33.889424 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:33.889434 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:33.889450 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:33.889939 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:33.890272 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:33.890277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:33.890281 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:33.890451 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:33.890964 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:33.890972 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:33.891293 (+ 321us) client_negotiation.cc:769] Sending connection context
0618 22:39:33.891527 (+ 234us) client_negotiation.cc:240] Negotiation successful
0618 22:39:33.892014 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":285,"threads_started":1}
I20260618 22:39:33.894259 14724 heartbeater.cc:347] Connected to a master server at 127.5.213.254:43463
I20260618 22:39:33.894562 14724 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:33.895401 14724 heartbeater.cc:511] Master 127.5.213.254:43463 requested a full tablet report, sending...
I20260618 22:39:33.896755 14268 ts_manager.cc:194] Registered new tserver with Master: bfc429e81766413dba69afbfc6fa1871 (127.5.213.195:43741)
I20260618 22:39:33.897516 14268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:44129
I20260618 22:39:33.901733 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.193.keytab.
Entry for principal oryx/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.193.keytab.
I20260618 22:39:33.943289 14457 heartbeater.cc:503] Master 127.5.213.254:43463 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.5.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.194.keytab.
Entry for principal oryx/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.194.keytab.
WARNING: no policy specified for oryx/127.5.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.195.keytab.
Entry for principal oryx/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.195.keytab.
I20260618 22:39:34.019222 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 14329
Jun 18 22:39:34 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:39:34.034610 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:34.020262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34558 (local address 127.5.213.193:43995)
0618 22:39:34.021407 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:34.021414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:34.021439 (+ 25us) server_negotiation.cc:407] Connection header received
0618 22:39:34.021499 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:34.021505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:34.021712 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:34.021839 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:34.023144 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:34.024128 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:34.025420 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:34.025954 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:34.026045 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:34.029728 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:34.029739 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:34.029753 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:34.029792 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:34.032478 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:34.033047 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:34.033053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:34.033056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:34.033169 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:34.033425 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:34.033428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:34.033430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:34.033732 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:34.033938 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:34.034134 (+ 196us) server_negotiation.cc:299] Negotiation successful
0618 22:39:34.034334 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":986,"thread_start_us":830,"threads_started":1}
I20260618 22:39:34.035866 14452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:34.115309 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 14329
I20260618 22:39:34.128376 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 14463
Jun 18 22:39:34 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.194@KRBTEST.COM
I20260618 22:39:34.145828 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:34.129224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46230 (local address 127.5.213.194:41549)
0618 22:39:34.129701 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:34.129708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:34.129732 (+ 24us) server_negotiation.cc:407] Connection header received
0618 22:39:34.129790 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:34.129796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:34.129998 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:34.130134 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:34.131352 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:34.132336 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:34.133606 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:34.134136 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:34.134256 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:34.138086 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:34.138097 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:34.138111 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:34.138151 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:34.143594 (+ 5443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:34.144229 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:34.144237 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:34.144239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:34.144356 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:34.144621 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:34.144624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:34.144626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:34.144938 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:34.145148 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:34.145355 (+ 207us) server_negotiation.cc:299] Negotiation successful
0618 22:39:34.145542 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":227,"threads_started":1}
I20260618 22:39:34.147053 14586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:34.226303 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 14463
I20260618 22:39:34.238294 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 14596
Jun 18 22:39:34 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.195@KRBTEST.COM
I20260618 22:39:34.254040 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:34.239346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36082 (local address 127.5.213.195:43741)
0618 22:39:34.239950 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:34.239957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:34.239984 (+ 27us) server_negotiation.cc:407] Connection header received
0618 22:39:34.240041 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:34.240047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:34.240260 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:34.240404 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:34.241653 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:34.242748 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:34.243999 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:34.244544 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:34.244654 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:34.248758 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:34.248777 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:34.248792 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:34.248823 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:34.251402 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:34.252079 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:34.252087 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:34.252089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:34.252209 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:34.252568 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:34.252572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:34.252574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:34.252920 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:34.253209 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:34.253479 (+ 270us) server_negotiation.cc:299] Negotiation successful
0618 22:39:34.253716 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":296,"threads_started":1}
I20260618 22:39:34.255396 14719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:34.335312 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 14596
I20260618 22:39:34.347456 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 14236
I20260618 22:39:34.348551 14298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:34.446895 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 14236
I20260618 22:39:34.462778 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:43463
--webserver_interface=127.5.213.254
--webserver_port=39677
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:43463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:34.772541 14741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:34.772862 14741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:34.772994 14741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:34.780112 14741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:39:34.780233 14741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:34.780311 14741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:34.780383 14741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:39:34.780444 14741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:39:34.790153 14741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:43463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:43463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=39677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:34.791445 14741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:34.792742 14741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:34.803913 14747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:34.803978 14749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:34.803948 14746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:34.804664 14741 server_base.cc:1034] running on GCE node
I20260618 22:39:34.805390 14741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:34.806682 14741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:34.807883 14741 hybrid_clock.cc:648] HybridClock initialized: now 1781822374807827 us; error 45 us; skew 500 ppm
I20260618 22:39:34.808187 14741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:34 dist-test-slave-lnj3 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822374, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:34.810992 14741 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:39:34.812302 14741 webserver.cc:460] Webserver started at http://127.5.213.254:39677/ using document root <none> and password file <none>
I20260618 22:39:34.812925 14741 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:34.813031 14741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:34.818082 14741 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260618 22:39:34.820714 14756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:34.821935 14741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:39:34.822069 14741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "2dd472574dde4b2f9168813488669463"
format_stamp: "Formatted at 2026-06-18 22:39:32 on dist-test-slave-lnj3"
server_key: "70632e054d6ad306a5cfd35825766733"
server_key_iv: "c94be0f31b5b8e5377359fb2f498d478"
server_key_version: "encryptionkey@0"
I20260618 22:39:34.823048 14741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:34.844101 14741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:34.848151 14741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:34.848387 14741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:34.864966 14741 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:43463
I20260618 22:39:34.864987 14807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:43463 every 8 connection(s)
I20260618 22:39:34.866459 14741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:39:34.871047 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 14741
I20260618 22:39:34.871753 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:43995
--local_ip_for_outbound_sockets=127.5.213.193
--tserver_master_addrs=127.5.213.254:43463
--webserver_port=43163
--webserver_interface=127.5.213.193
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.213.193
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:39:34.873139 14808 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:39:34.876327 14808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463: Bootstrap starting.
I20260618 22:39:34.894412 14808 log.cc:824] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:34.902989 14808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:39:34.903471 14808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463: Bootstrap complete.
I20260618 22:39:34.912160 14808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd472574dde4b2f9168813488669463" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43463 } }
I20260618 22:39:34.912974 14808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd472574dde4b2f9168813488669463, State: Initialized, Role: FOLLOWER
I20260618 22:39:34.913604 14808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd472574dde4b2f9168813488669463" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43463 } }
I20260618 22:39:34.913808 14808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:39:34.913918 14808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:39:34.914069 14808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [term 1 FOLLOWER]: Advancing to term 2
I20260618 22:39:34.917752 14808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd472574dde4b2f9168813488669463" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43463 } }
I20260618 22:39:34.918243 14808 leader_election.cc:302] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dd472574dde4b2f9168813488669463; no voters:
I20260618 22:39:34.918830 14808 leader_election.cc:288] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260618 22:39:34.919016 14812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [term 2 FOLLOWER]: Leader election won for term 2
I20260618 22:39:34.919939 14812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [term 2 LEADER]: Becoming Leader. State: Replica: 2dd472574dde4b2f9168813488669463, State: Running, Role: LEADER
I20260618 22:39:34.920537 14812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd472574dde4b2f9168813488669463" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43463 } }
I20260618 22:39:34.920912 14808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:39:34.922948 14813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2dd472574dde4b2f9168813488669463" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd472574dde4b2f9168813488669463" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43463 } } }
I20260618 22:39:34.923297 14813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:34.925455 14814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dd472574dde4b2f9168813488669463. Latest consensus state: current_term: 2 leader_uuid: "2dd472574dde4b2f9168813488669463" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd472574dde4b2f9168813488669463" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43463 } } }
I20260618 22:39:34.925832 14814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:34.927397 14817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:39:34.934968 14817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:39:34.936488 14817 catalog_manager.cc:1259] Loaded cluster ID: f04b1f5ea0c34a0c90e6702b33ed7471
I20260618 22:39:34.936669 14817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:39:34.940517 14817 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:39:34.941993 14817 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2dd472574dde4b2f9168813488669463: Loaded TSK: 0
I20260618 22:39:34.943544 14817 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 22:39:35.193389 14810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:35.193672 14810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:35.193801 14810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:35.201226 14810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:35.201352 14810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:35.201505 14810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:39:35.211293 14810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.193.keytab
--principal=oryx/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:43995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=43163
--tserver_master_addrs=127.5.213.254:43463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:35.212591 14810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:35.213865 14810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:35.227348 14834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:35.227324 14835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:35.227988 14837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:35.230943 14810 server_base.cc:1034] running on GCE node
I20260618 22:39:35.231565 14810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:35.232590 14810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:35.233834 14810 hybrid_clock.cc:648] HybridClock initialized: now 1781822375233786 us; error 39 us; skew 500 ppm
I20260618 22:39:35.234160 14810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:35 dist-test-slave-lnj3 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822375, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:35.237404 14810 init.cc:377] Logged in from keytab as oryx/127.5.213.193@KRBTEST.COM (short username oryx)
I20260618 22:39:35.238961 14810 webserver.cc:460] Webserver started at http://127.5.213.193:43163/ using document root <none> and password file <none>
I20260618 22:39:35.239638 14810 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:35.239756 14810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:35.245168 14810 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260618 22:39:35.248003 14844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:35.249382 14810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:39:35.249519 14810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "4e8745d01e2c4fb98ba3323b05dbe9ec"
format_stamp: "Formatted at 2026-06-18 22:39:32 on dist-test-slave-lnj3"
server_key: "174d6506b397abc31bba3087872c3584"
server_key_iv: "efd01f50a12a3ae20e9b26efcc073d18"
server_key_version: "encryptionkey@0"
I20260618 22:39:35.250450 14810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:35.304613 14810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:35.308892 14810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:35.309145 14810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:35.310123 14810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:35.311439 14810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:35.311533 14810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:35.311627 14810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:35.311678 14810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:35.341364 14810 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:43995
I20260618 22:39:35.341434 14956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:43995 every 8 connection(s)
I20260618 22:39:35.342839 14810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:39:35.352300 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 14810
I20260618 22:39:35.352933 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:41549
--local_ip_for_outbound_sockets=127.5.213.194
--tserver_master_addrs=127.5.213.254:43463
--webserver_port=42201
--webserver_interface=127.5.213.194
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.213.194
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:39:35 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:35 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
W20260618 22:39:35.680075 14961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:35.680377 14961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:35.680516 14961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:35.688059 14961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:35.688184 14961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:35.688355 14961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:39:35.698236 14961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.194.keytab
--principal=oryx/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:41549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=42201
--tserver_master_addrs=127.5.213.254:43463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:35.699558 14961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:35.700875 14961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:35.714979 14968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:35.715441 14967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:35.715687 14970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:35.716092 14961 server_base.cc:1034] running on GCE node
I20260618 22:39:35.716754 14961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:35.717738 14961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:35.718961 14961 hybrid_clock.cc:648] HybridClock initialized: now 1781822375718901 us; error 51 us; skew 500 ppm
I20260618 22:39:35.719256 14961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:35 dist-test-slave-lnj3 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822375, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:35.722528 14961 init.cc:377] Logged in from keytab as oryx/127.5.213.194@KRBTEST.COM (short username oryx)
I20260618 22:39:35.724123 14961 webserver.cc:460] Webserver started at http://127.5.213.194:42201/ using document root <none> and password file <none>
I20260618 22:39:35.724785 14961 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:35.724910 14961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:35.730716 14961 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260618 22:39:35.733461 14977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:35.734954 14961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:39:35.735098 14961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "bb06623180d74f53bc40f82927ce8c76"
format_stamp: "Formatted at 2026-06-18 22:39:33 on dist-test-slave-lnj3"
server_key: "c260f0cf37dbddef32ce21c95594863a"
server_key_iv: "20dd768744caa8a484999bb35ba65ea2"
server_key_version: "encryptionkey@0"
I20260618 22:39:35.735982 14961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:35.749768 14961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:35.753717 14961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:35.753947 14961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:35.754956 14961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:35.756237 14961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:35.756352 14961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:35.756451 14961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:35.756520 14961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:35.788997 14961 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:41549
I20260618 22:39:35.789040 15089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:41549 every 8 connection(s)
I20260618 22:39:35.790448 14961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:39:35.798652 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 14961
I20260618 22:39:35.799446 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:43741
--local_ip_for_outbound_sockets=127.5.213.195
--tserver_master_addrs=127.5.213.254:43463
--webserver_port=39463
--webserver_interface=127.5.213.195
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.213.195
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:39:35 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:35 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
W20260618 22:39:36.130033 15094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:36.130334 15094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:36.130474 15094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:36.137765 15094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:36.137895 15094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:36.138067 15094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:39:36.148030 15094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/krb5kdc/oryx_127.5.213.195.keytab
--principal=oryx/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:43741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=39463
--tserver_master_addrs=127.5.213.254:43463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:36.149335 15094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:36.150678 15094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:36.165771 15101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:36.167143 15100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:36.174010 15103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:36.174479 15094 server_base.cc:1034] running on GCE node
I20260618 22:39:36.175139 15094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:36.176128 15094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:36.177371 15094 hybrid_clock.cc:648] HybridClock initialized: now 1781822376177311 us; error 53 us; skew 500 ppm
I20260618 22:39:36.177676 15094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:36 dist-test-slave-lnj3 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822376, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:36.180634 15094 init.cc:377] Logged in from keytab as oryx/127.5.213.195@KRBTEST.COM (short username oryx)
I20260618 22:39:36.183809 15094 webserver.cc:460] Webserver started at http://127.5.213.195:39463/ using document root <none> and password file <none>
I20260618 22:39:36.184514 15094 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:36.184643 15094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:36.189859 15094 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.001s
I20260618 22:39:36.192584 15110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:36.193974 15094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:39:36.194121 15094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "bfc429e81766413dba69afbfc6fa1871"
format_stamp: "Formatted at 2026-06-18 22:39:33 on dist-test-slave-lnj3"
server_key: "3b385f05e1fcf5e027a5eb1d1205715a"
server_key_iv: "e34f965bbc60d1754b57c6cc38b3cf4e"
server_key_version: "encryptionkey@0"
I20260618 22:39:36.195078 15094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:36.220155 15094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:36.224567 15094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:36.224808 15094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:36.225838 15094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:36.227286 15094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:36.227388 15094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:36.227491 15094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:36.227559 15094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:36.326839 15094 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:43741
I20260618 22:39:36.326920 15222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:43741 every 8 connection(s)
I20260618 22:39:36.329087 15094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:39:36.336756 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 15094
Jun 18 22:39:36 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:36 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:36.370666 15228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:36.339579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57116 (local address 127.5.213.254:43463)
0618 22:39:36.343086 (+ 3507us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:36.343092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:36.343109 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:39:36.343160 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:36.343165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:36.343228 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:36.348093 (+ 4865us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:36.348360 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.349761 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.353141 (+ 3380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.353826 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.353951 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:36.357962 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:36.357974 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:36.358001 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:36.358052 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:36.364503 (+ 6451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:36.365530 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:36.365541 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:36.365553 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:36.365815 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:36.366276 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:36.366281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:36.366285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:36.367017 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:36.367354 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:36.368911 (+ 1557us) server_negotiation.cc:299] Negotiation successful
0618 22:39:36.369431 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3365,"thread_start_us":11087,"threads_started":1}
W20260618 22:39:36.492018 14960 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.136s user 0.004s sys 0.007s
W20260618 22:39:36.493301 14959 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:35.345754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:42515 (local address 127.5.213.254:43463)
0618 22:39:35.346513 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:35.346529 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:35.346816 (+ 287us) server_negotiation.cc:407] Connection header received
0618 22:39:35.348803 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:35.348817 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:35.349143 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:35.349468 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:35.351009 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:35.352519 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:35.354205 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:35.355249 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:35.355408 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:36.493005 (+1137597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.193:42515: BlockingRecv error: recv got EOF from 127.5.213.193:42515 (error 108)
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":283,"threads_started":1}
I20260618 22:39:36.493842 14960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:35.345867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.193:42515)
0618 22:39:35.346666 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:35.346704 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:35.348285 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:35.349630 (+ 1345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:35.349643 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:35.350255 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:35.350840 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:35.350860 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:35.352672 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:35.352676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:35.354081 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:35.354093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:35.355430 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:35.355435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:35.355629 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:35.356237 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:35.356259 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:36.492746 (+1136487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":307,"threads_started":1}
W20260618 22:39:36.494649 14957 heartbeater.cc:644] Failed to heartbeat to 127.5.213.254:43463 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.213.254:43463: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Jun 18 22:39:36 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:36 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:36.504634 14960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:36.495505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.193:56653)
0618 22:39:36.495755 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:36.495773 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:36.495878 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:36.496168 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:36.496173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:36.496620 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:36.496815 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:36.496827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.498366 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.498371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:36.499573 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:36.499584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.500458 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.500464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:36.500618 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:36.501174 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:36.501192 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:36.504368 (+ 3176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136}
W20260618 22:39:36.504793 14959 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:36.495674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:56653 (local address 127.5.213.254:43463)
0618 22:39:36.495811 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:36.495816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:36.495833 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:39:36.495984 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:36.495988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:36.496042 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:36.496118 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:36.497116 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.497919 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.499730 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.500288 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.500395 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:36.504575 (+ 4180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.193:56653: BlockingRecv error: recv got EOF from 127.5.213.193:56653 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 18 22:39:36 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:36 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:36.514379 14960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:36.505445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.193:58333)
0618 22:39:36.505765 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:36.505782 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:36.505881 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:36.506181 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:36.506186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:36.506467 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:36.506601 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:36.506608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.507583 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.507589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:36.508508 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:36.508519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.509170 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.509176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:36.509353 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:36.509870 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:36.509887 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:36.514111 (+ 4224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228}
W20260618 22:39:36.514508 14959 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:36.505615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:58333 (local address 127.5.213.254:43463)
0618 22:39:36.505760 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:36.505764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:36.505827 (+ 63us) server_negotiation.cc:407] Connection header received
0618 22:39:36.505978 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:36.505983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:36.506055 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:36.506130 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:36.506735 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.507456 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.508637 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.509034 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.509133 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:36.514289 (+ 5156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.193:58333: BlockingRecv error: recv got EOF from 127.5.213.193:58333 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260618 22:39:36.514978 14957 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 22:39:36.967103 15093 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.164s user 0.006s sys 0.007s
W20260618 22:39:36.968569 15092 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:35.793242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:38009 (local address 127.5.213.254:43463)
0618 22:39:35.793625 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:35.793630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:35.795116 (+ 1486us) server_negotiation.cc:407] Connection header received
0618 22:39:35.796927 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:35.796933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:35.797011 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:35.797103 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:35.798620 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:35.799430 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:35.801203 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:35.801619 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:35.801727 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:36.968163 (+1166436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.194:38009: BlockingRecv error: recv got EOF from 127.5.213.194:38009 (error 108)
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":190,"threads_started":1}
I20260618 22:39:36.968873 15093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:35.794101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.194:38009)
0618 22:39:35.794956 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:35.795004 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:35.796655 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:35.797354 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:35.797363 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:35.797892 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:35.798453 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:35.798470 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:35.799588 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:35.799595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:35.801065 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:35.801081 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:35.801840 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:35.801847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:35.802066 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:35.802717 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:35.802740 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:36.967792 (+1165052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":334,"threads_started":1}
W20260618 22:39:36.969792 15090 heartbeater.cc:644] Failed to heartbeat to 127.5.213.254:43463 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.213.254:43463: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Jun 18 22:39:36 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:36 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
W20260618 22:39:36.981194 15092 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:36.970673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:58771 (local address 127.5.213.254:43463)
0618 22:39:36.970801 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:36.970805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:36.970820 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:39:36.970903 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:36.970906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:36.970962 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:36.971034 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:36.972050 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.972886 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.974851 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.975237 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.975324 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:36.980952 (+ 5628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.194:58771: BlockingRecv error: recv got EOF from 127.5.213.194:58771 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260618 22:39:36.981317 15093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:36.970387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.194:58771)
0618 22:39:36.970618 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:36.970659 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:36.970789 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:36.971082 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:36.971087 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:36.971652 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:36.971878 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:36.971891 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.973024 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.973030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:36.974711 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:36.974728 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.975398 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.975405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:36.975588 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:36.976255 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:36.976298 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:36.980744 (+ 4446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
Jun 18 22:39:36 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:36 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:36.990671 15093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:36.982234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.194:33721)
0618 22:39:36.982616 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:36.982653 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:36.982757 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:36.983103 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:36.983113 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:36.983404 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:36.983549 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:36.983557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.984537 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.984543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:36.985525 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:36.985536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.986181 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.986187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:36.986349 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:36.987000 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:36.987016 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:36.990369 (+ 3353us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285}
W20260618 22:39:36.990829 15092 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:36.982450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:33721 (local address 127.5.213.254:43463)
0618 22:39:36.982597 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:36.982602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:36.982698 (+ 96us) server_negotiation.cc:407] Connection header received
0618 22:39:36.982864 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:36.982869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:36.982927 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:36.983002 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:36.983685 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.984400 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.985649 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.986042 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.986145 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:36.990567 (+ 4422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.194:33721: BlockingRecv error: recv got EOF from 127.5.213.194:33721 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260618 22:39:36.991245 15090 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 22:39:37.408509 15225 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.047s user 0.000s sys 0.018s
W20260618 22:39:37.409560 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:36.334216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:50825 (local address 127.5.213.254:43463)
0618 22:39:36.339634 (+ 5418us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:36.339641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:36.339665 (+ 24us) server_negotiation.cc:407] Connection header received
0618 22:39:36.342247 (+ 2582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:36.342254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:36.342367 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:36.342475 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:36.351684 (+ 9209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.353000 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.356833 (+ 3833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.357503 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.357595 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:37.409365 (+1051770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.195:50825: BlockingRecv error: recv got EOF from 127.5.213.195:50825 (error 108)
Metrics: {"server-negotiator.queue_time_us":5287,"thread_start_us":5165,"threads_started":1}
I20260618 22:39:37.410053 15225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:36.333470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.195:50825)
0618 22:39:36.336033 (+ 2563us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:36.336103 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:36.341756 (+ 5653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:36.343725 (+ 1969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:36.343740 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:36.344615 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:36.345592 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:36.345622 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.354271 (+ 8649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.354278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:36.356630 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:36.356649 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:36.359132 (+ 2483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:36.359140 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:36.359494 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:36.360985 (+ 1491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:36.361019 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:37.409190 (+1048171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2169,"thread_start_us":1876,"threads_started":1}
W20260618 22:39:37.410919 15223 heartbeater.cc:644] Failed to heartbeat to 127.5.213.254:43463 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.213.254:43463: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Jun 18 22:39:37 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:37 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:37.419409 15225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:37.411485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.195:52795)
0618 22:39:37.411664 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:37.411684 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:37.411789 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:37.412031 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:37.412035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:37.412356 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:37.412504 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:37.412511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:37.413492 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:37.413496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:37.414427 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:37.414438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:37.415199 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:37.415204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:37.415344 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:37.415868 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:37.415885 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:37.419145 (+ 3260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260618 22:39:37.419540 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:37.411560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:52795 (local address 127.5.213.254:43463)
0618 22:39:37.411696 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:37.411700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:37.411732 (+ 32us) server_negotiation.cc:407] Connection header received
0618 22:39:37.411872 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:37.411875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:37.411930 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:37.412007 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:37.412627 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:37.413367 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:37.414604 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:37.415036 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:37.415152 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:37.419344 (+ 4192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.195:52795: BlockingRecv error: recv got EOF from 127.5.213.195:52795 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 18 22:39:37 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:37 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:37.428171 15225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:37.420200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.195:46273)
0618 22:39:37.420323 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:37.420338 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:37.420416 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:37.420848 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:37.420853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:37.421132 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:37.421288 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:37.421296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:37.422234 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:37.422238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:37.423197 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:37.423208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:37.423916 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:37.423922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:37.424081 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:37.424611 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:37.424628 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:37.427975 (+ 3347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260618 22:39:37.428491 15223 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 22:39:37.428632 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:37.420316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:46273 (local address 127.5.213.254:43463)
0618 22:39:37.420570 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:37.420574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:37.420587 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:39:37.420624 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:37.420627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:37.420719 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:37.420798 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:37.421430 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:37.422108 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:37.423319 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:37.423780 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:37.423884 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:37.428229 (+ 4345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.195:46273: BlockingRecv error: recv got EOF from 127.5.213.195:46273 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Jun 18 22:39:37 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:37 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:37.523931 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:37.515504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.193:48183)
0618 22:39:37.515981 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:37.515997 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:37.516109 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:37.516388 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:37.516392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:37.516800 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:37.516941 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:37.516950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:37.517870 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:37.517873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:37.518843 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:37.518854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:37.519524 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:37.519528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:37.519654 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:37.520198 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:37.520214 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:37.523710 (+ 3496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":275,"threads_started":1}
W20260618 22:39:37.524087 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:37.515581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:48183 (local address 127.5.213.254:43463)
0618 22:39:37.515793 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:37.515801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:37.516073 (+ 272us) server_negotiation.cc:407] Connection header received
0618 22:39:37.516202 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:37.516206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:37.516265 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:37.516353 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:37.517074 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:37.517748 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:37.518959 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:37.519378 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:37.519468 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:37.523897 (+ 4429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.193:48183: BlockingRecv error: recv got EOF from 127.5.213.193:48183 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 18 22:39:37 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:37 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:38.000437 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:37.991766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.194:37089)
0618 22:39:37.992226 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:37.992245 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:37.992352 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:37.992650 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:37.992654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:37.993061 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:37.993202 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:37.993212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:37.994163 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:37.994166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:37.995316 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:37.995330 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:37.996140 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:37.996146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:37.996330 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:37.996870 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:37.996888 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:38.000207 (+ 3319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":257,"threads_started":1}
W20260618 22:39:38.000581 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:37.991834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:37089 (local address 127.5.213.254:43463)
0618 22:39:37.992032 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:37.992039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:37.992312 (+ 273us) server_negotiation.cc:407] Connection header received
0618 22:39:37.992435 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:37.992438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:37.992496 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:37.992573 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:37.993324 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:37.994047 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:37.995453 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:37.995951 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:37.996063 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:38.000388 (+ 4325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.194:37089: BlockingRecv error: recv got EOF from 127.5.213.194:37089 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 18 22:39:38 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:38 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:38.438107 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:38.429110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.195:38055)
0618 22:39:38.429755 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:38.429776 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:38.429926 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:38.430242 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:38.430247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:38.430746 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:38.430896 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:38.430904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:38.431863 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:38.431867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:38.432819 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:38.432830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:38.433611 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:38.433618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:38.433766 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:38.434336 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:38.434356 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:38.437873 (+ 3517us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":311,"threads_started":1}
W20260618 22:39:38.438277 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:38.429203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:38055 (local address 127.5.213.254:43463)
0618 22:39:38.429399 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:38.429406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:38.429881 (+ 475us) server_negotiation.cc:407] Connection header received
0618 22:39:38.430052 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:38.430056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:38.430115 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:38.430195 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:38.431043 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:38.431729 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:38.432952 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:38.433427 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:38.433536 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:38.438100 (+ 4564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.195:38055: BlockingRecv error: recv got EOF from 127.5.213.195:38055 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 18 22:39:38 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:38 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:38.533056 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:38.524808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.193:49803)
0618 22:39:38.525215 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:38.525230 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:38.525364 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:38.525619 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:38.525622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:38.525979 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:38.526181 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:38.526190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:38.527231 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:38.527235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:38.528238 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:38.528250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:38.528850 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:38.528857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:38.528995 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:38.529567 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:38.529589 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:38.532827 (+ 3238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":224,"threads_started":1}
W20260618 22:39:38.533275 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:38.524883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:49803 (local address 127.5.213.254:43463)
0618 22:39:38.525091 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:38.525096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:38.525299 (+ 203us) server_negotiation.cc:407] Connection header received
0618 22:39:38.525451 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:38.525454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:38.525514 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:38.525591 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:38.526327 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:38.527102 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:38.528357 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:38.528728 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:38.528815 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:38.533035 (+ 4220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.193:49803: BlockingRecv error: recv got EOF from 127.5.213.193:49803 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 18 22:39:39 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:39 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:39.009580 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:39.001344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.194:59335)
0618 22:39:39.001838 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:39.001857 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:39.001987 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:39.002237 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:39.002240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:39.002592 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:39.002819 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:39.002829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:39.003776 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:39.003779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:39.004696 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:39.004708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:39.005320 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:39.005325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:39.005447 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:39.006099 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:39.006116 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:39.009329 (+ 3213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":286,"threads_started":1}
W20260618 22:39:39.009644 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:39.001405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:59335 (local address 127.5.213.254:43463)
0618 22:39:39.001570 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:39.001574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:39.001920 (+ 346us) server_negotiation.cc:407] Connection header received
0618 22:39:39.002071 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:39.002074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:39.002131 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:39.002203 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:39.002932 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:39.003662 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:39.004807 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:39.005190 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:39.005284 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:39.009478 (+ 4194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.194:59335: BlockingRecv error: recv got EOF from 127.5.213.194:59335 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 18 22:39:39 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:39 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:39.454308 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:39.438938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.195:40349)
0618 22:39:39.439554 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:39.439569 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:39.439688 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:39.439997 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:39.440000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:39.440419 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:39.440618 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:39.440629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:39.441530 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:39.441533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:39.442425 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:39.442436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:39.443025 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:39.443029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:39.443142 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:39.451026 (+ 7884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:39.451044 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:39.454055 (+ 3011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":276,"threads_started":1}
W20260618 22:39:39.454442 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:39.439077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:40349 (local address 127.5.213.254:43463)
0618 22:39:39.439284 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:39.439290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:39.439639 (+ 349us) server_negotiation.cc:407] Connection header received
0618 22:39:39.439816 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:39.439820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:39.439890 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:39.439965 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:39.440726 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:39.441434 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:39.442526 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:39.442921 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:39.443012 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:39.454241 (+ 11229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.195:40349: BlockingRecv error: recv got EOF from 127.5.213.195:40349 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 18 22:39:39 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:39 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:39.541759 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:39.533873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.193:41891)
0618 22:39:39.534372 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:39.534387 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:39.534497 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:39.534805 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:39.534808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:39.535123 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:39.535303 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:39.535311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:39.536218 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:39.536221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:39.537116 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:39.537126 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:39.537724 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:39.537728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:39.537842 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:39.538360 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:39.538376 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:39.541531 (+ 3155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":251,"threads_started":1}
W20260618 22:39:39.541954 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:39.533939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:41891 (local address 127.5.213.254:43463)
0618 22:39:39.534081 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:39.534084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:39.534465 (+ 381us) server_negotiation.cc:407] Connection header received
0618 22:39:39.534581 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:39.534584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:39.534688 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:39.534775 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:39.535410 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:39.536113 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:39.537219 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:39.537622 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:39.537710 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:39.541734 (+ 4024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.193:41891: BlockingRecv error: recv got EOF from 127.5.213.193:41891 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 18 22:39:40 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:40 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:40.019013 15237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:40.010412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.194:55839)
0618 22:39:40.010967 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:40.010990 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:40.011146 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:40.011524 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:40.011529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:40.011954 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:40.012154 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:40.012167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:40.013172 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:40.013178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:40.014084 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:40.014096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:40.014796 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:40.014802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:40.014978 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:40.015516 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:40.015533 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:40.018789 (+ 3256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":254,"threads_started":1}
W20260618 22:39:40.019205 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:40.010533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:55839 (local address 127.5.213.254:43463)
0618 22:39:40.010765 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:40.010771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:40.011082 (+ 311us) server_negotiation.cc:407] Connection header received
0618 22:39:40.011285 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:40.011290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:40.011369 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:40.011460 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:40.012279 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:40.013034 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:40.014218 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:40.014667 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:40.014789 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:40.018976 (+ 4187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.194:55839: BlockingRecv error: recv got EOF from 127.5.213.194:55839 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Jun 18 22:39:40 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:40 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:40.463505 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:40.455154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.195:52823)
0618 22:39:40.455609 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:40.455624 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:40.455734 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:40.456131 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:40.456134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:40.456483 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:40.456690 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:40.456699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:40.457762 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:40.457766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:40.458712 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:40.458724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:40.459385 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:40.459389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:40.459499 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:40.460095 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:40.460113 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:40.463201 (+ 3088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":213,"threads_started":1}
W20260618 22:39:40.463641 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:40.455286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:52823 (local address 127.5.213.254:43463)
0618 22:39:40.455491 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:40.455496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:40.455693 (+ 197us) server_negotiation.cc:407] Connection header received
0618 22:39:40.455912 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:40.455917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:40.456003 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:40.456103 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:40.456834 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:40.457625 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:40.458845 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:40.459278 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:40.459380 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:40.463415 (+ 4035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.195:52823: BlockingRecv error: recv got EOF from 127.5.213.195:52823 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 18 22:39:40 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:40 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:40.551057 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:40.542565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.193:53581)
0618 22:39:40.543064 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:40.543079 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:40.543201 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:40.543502 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:40.543506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:40.543885 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:40.544039 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:40.544048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:40.545035 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:40.545039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:40.546175 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:40.546186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:40.546969 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:40.546975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:40.547148 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:40.547694 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:40.547711 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:40.550730 (+ 3019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":295,"threads_started":1}
W20260618 22:39:40.551172 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:40.542620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:53581 (local address 127.5.213.254:43463)
0618 22:39:40.542826 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:40.542832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:40.543152 (+ 320us) server_negotiation.cc:407] Connection header received
0618 22:39:40.543317 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:40.543322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:40.543384 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:40.543464 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:40.544176 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:40.544906 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:40.546291 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:40.546842 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:40.546944 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:40.550932 (+ 3988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.193:53581: BlockingRecv error: recv got EOF from 127.5.213.193:53581 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 18 22:39:41 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:41 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:41.028707 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:41.019982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.194:55827)
0618 22:39:41.020464 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:41.020485 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:41.020609 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:41.020966 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:41.020970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:41.021372 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:41.021526 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:41.021535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:41.022528 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:41.022533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:41.023619 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:41.023633 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:41.024412 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:41.024417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:41.024592 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:41.025209 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:41.025227 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:41.028462 (+ 3235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":271,"threads_started":1}
W20260618 22:39:41.028797 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:41.020054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:55827 (local address 127.5.213.254:43463)
0618 22:39:41.020233 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:41.020239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:41.020557 (+ 318us) server_negotiation.cc:407] Connection header received
0618 22:39:41.020743 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:41.020748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:41.020832 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:41.020919 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:41.021683 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:41.022397 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:41.023740 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:41.024278 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:41.024387 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:41.028611 (+ 4224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.194:55827: BlockingRecv error: recv got EOF from 127.5.213.194:55827 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 18 22:39:41 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:41 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:41.472651 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:41.464461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.195:33895)
0618 22:39:41.464935 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:41.464950 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:41.465071 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:41.465427 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:41.465432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:41.465858 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:41.466028 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:41.466038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:41.467053 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:41.467057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:41.468091 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:41.468104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:41.468739 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:41.468744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:41.468894 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:41.469394 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:41.469410 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:41.472418 (+ 3008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":258,"threads_started":1}
W20260618 22:39:41.472811 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:41.464521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:33895 (local address 127.5.213.254:43463)
0618 22:39:41.464689 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:41.464695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:41.465029 (+ 334us) server_negotiation.cc:407] Connection header received
0618 22:39:41.465213 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:41.465217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:41.465285 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:41.465369 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:41.466170 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:41.466923 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:41.468219 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:41.468608 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:41.468712 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:41.472595 (+ 3883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.195:33895: BlockingRecv error: recv got EOF from 127.5.213.195:33895 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 18 22:39:41 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:41 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:41.559700 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:41.551885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.193:58109)
0618 22:39:41.552340 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:41.552355 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:41.552463 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:41.552705 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:41.552708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:41.553094 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:41.553267 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:41.553276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:41.554224 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:41.554228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:41.555228 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:41.555241 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:41.555825 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:41.555829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:41.555951 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:41.556480 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:41.556496 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:41.559485 (+ 2989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":224,"threads_started":1}
W20260618 22:39:41.559876 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:41.551937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:58109 (local address 127.5.213.254:43463)
0618 22:39:41.552086 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:41.552089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:41.552414 (+ 325us) server_negotiation.cc:407] Connection header received
0618 22:39:41.552541 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:41.552544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:41.552600 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:41.552679 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:41.553372 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:41.554112 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:41.555340 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:41.555719 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:41.555806 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:41.559693 (+ 3887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.193:58109: BlockingRecv error: recv got EOF from 127.5.213.193:58109 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 18 22:39:42 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:42 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:42.037757 15243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:42.029539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.194:43749)
0618 22:39:42.030059 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:42.030080 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:42.030198 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:42.030456 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:42.030459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:42.030916 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:42.031066 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:42.031074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:42.032027 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:42.032030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:42.032943 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:42.032953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:42.033555 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:42.033559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:42.033690 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:42.034204 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:42.034220 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:42.037528 (+ 3308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":302,"threads_started":1}
W20260618 22:39:42.038007 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:42.029686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:43749 (local address 127.5.213.254:43463)
0618 22:39:42.029851 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:42.029855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:42.030141 (+ 286us) server_negotiation.cc:407] Connection header received
0618 22:39:42.030288 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:42.030291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:42.030349 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:42.030424 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:42.031177 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:42.031930 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:42.033045 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:42.033447 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:42.033532 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:42.037768 (+ 4236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.194:43749: BlockingRecv error: recv got EOF from 127.5.213.194:43749 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 18 22:39:42 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:42 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:42.481742 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:42.473569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.195:40303)
0618 22:39:42.474204 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:42.474220 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:42.474400 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:42.474766 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:42.474771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:42.475179 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:42.475336 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:42.475344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:42.476320 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:42.476323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:42.477199 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:42.477208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:42.477807 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:42.477810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:42.477919 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:42.478481 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:42.478498 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:42.481518 (+ 3020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":399,"threads_started":1}
W20260618 22:39:42.481905 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:42.473649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:40303 (local address 127.5.213.254:43463)
0618 22:39:42.473857 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:42.473864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:42.474302 (+ 438us) server_negotiation.cc:407] Connection header received
0618 22:39:42.474535 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:42.474540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:42.474612 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:42.474740 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:42.475457 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:42.476201 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:42.477308 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:42.477708 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:42.477792 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:42.481703 (+ 3911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.195:40303: BlockingRecv error: recv got EOF from 127.5.213.195:40303 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 18 22:39:42 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:42 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:42.569192 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:42.560497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.193:59703)
0618 22:39:42.560939 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:42.560956 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:42.561072 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:42.561374 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:42.561378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:42.561825 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:42.561987 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:42.561996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:42.562985 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:42.562990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:42.563936 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:42.563947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:42.564607 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:42.564611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:42.564758 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:42.565288 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:42.565305 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:42.568904 (+ 3599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":261,"threads_started":1}
W20260618 22:39:42.569331 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:42.560628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:59703 (local address 127.5.213.254:43463)
0618 22:39:42.560861 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:42.560868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:42.561037 (+ 169us) server_negotiation.cc:407] Connection header received
0618 22:39:42.561160 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:42.561163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:42.561221 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:42.561312 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:42.562105 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:42.562844 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:42.564042 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:42.564442 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:42.564530 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:42.569110 (+ 4580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.193:59703: BlockingRecv error: recv got EOF from 127.5.213.193:59703 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 18 22:39:43 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:43 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:43.046904 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:43.038717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.194:57033)
0618 22:39:43.039247 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:43.039265 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:43.039377 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:43.039630 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:43.039633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:43.040020 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:43.040228 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:43.040238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:43.041179 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:43.041183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:43.042090 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:43.042102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:43.042780 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:43.042785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:43.042911 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:43.043464 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:43.043480 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:43.046559 (+ 3079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":291,"threads_started":1}
W20260618 22:39:43.046926 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:43.038811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:57033 (local address 127.5.213.254:43463)
0618 22:39:43.038992 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:43.038998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:43.039338 (+ 340us) server_negotiation.cc:407] Connection header received
0618 22:39:43.039461 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:43.039464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:43.039525 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:43.039600 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:43.040354 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:43.041064 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:43.042201 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:43.042591 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:43.042744 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:43.046774 (+ 4030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.194:57033: BlockingRecv error: recv got EOF from 127.5.213.194:57033 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 18 22:39:43 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:43 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:43.490931 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:43.482755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.195:34125)
0618 22:39:43.483150 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:43.483167 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:43.483297 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:43.483626 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:43.483630 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:43.484047 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:43.484202 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:43.484213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:43.485448 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:43.485452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:43.486356 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:43.486367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:43.486980 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:43.486985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:43.487101 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:43.487628 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:43.487645 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:43.490707 (+ 3062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":202,"threads_started":1}
W20260618 22:39:43.491036 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:43.482818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:34125 (local address 127.5.213.254:43463)
0618 22:39:43.483013 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:43.483018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:43.483237 (+ 219us) server_negotiation.cc:407] Connection header received
0618 22:39:43.483400 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:43.483404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:43.483481 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:43.483583 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:43.484348 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:43.485329 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:43.486461 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:43.486869 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:43.486956 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:43.490851 (+ 3895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.195:34125: BlockingRecv error: recv got EOF from 127.5.213.195:34125 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 18 22:39:43 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:43 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:43.578433 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:43.570048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.193:37573)
0618 22:39:43.570548 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:43.570564 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:43.570707 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:43.570976 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:43.570979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:43.571348 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:43.571529 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:43.571538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:43.572588 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:43.572591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:43.573509 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:43.573519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:43.574115 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:43.574119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:43.574225 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:43.574755 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:43.574777 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:43.578216 (+ 3439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":224,"threads_started":1}
W20260618 22:39:43.578523 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:43.570111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:37573 (local address 127.5.213.254:43463)
0618 22:39:43.570315 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:43.570321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:43.570660 (+ 339us) server_negotiation.cc:407] Connection header received
0618 22:39:43.570798 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:43.570801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:43.570872 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:43.570947 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:43.571648 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:43.572477 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:43.573610 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:43.574014 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:43.574096 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:43.578359 (+ 4263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.193:37573: BlockingRecv error: recv got EOF from 127.5.213.193:37573 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 18 22:39:44 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:44 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:44.056483 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:44.047801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.194:41535)
0618 22:39:44.048413 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:44.048433 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:44.048545 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:44.048809 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:44.048812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:44.049199 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:44.049374 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:44.049383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:44.050436 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:44.050441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:44.051400 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:44.051413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:44.052123 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:44.052127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:44.052258 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:44.052820 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:44.052837 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:44.056163 (+ 3326us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":296,"threads_started":1}
W20260618 22:39:44.056619 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:44.047801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:41535 (local address 127.5.213.254:43463)
0618 22:39:44.047997 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:44.048004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:44.048526 (+ 522us) server_negotiation.cc:407] Connection header received
0618 22:39:44.048627 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:44.048631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:44.048695 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:44.048774 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:44.049535 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:44.050298 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:44.051542 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:44.051983 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:44.052079 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:44.056386 (+ 4307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.194:41535: BlockingRecv error: recv got EOF from 127.5.213.194:41535 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 18 22:39:44 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:44 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:44.507035 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:44.491844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.195:59917)
0618 22:39:44.492491 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:44.492507 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:44.492632 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:44.492916 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:44.492920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:44.493289 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:44.493450 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:44.493457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:44.494422 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:44.494426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:44.495594 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:44.495612 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:44.496375 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:44.496381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:44.496535 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:44.503360 (+ 6825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:44.503379 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:44.506730 (+ 3351us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":347,"threads_started":1}
W20260618 22:39:44.507090 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:44.491921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:59917 (local address 127.5.213.254:43463)
0618 22:39:44.492071 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:44.492075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:44.492584 (+ 509us) server_negotiation.cc:407] Connection header received
0618 22:39:44.492722 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:44.492726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:44.492794 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:44.492875 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:44.493563 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:44.494292 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:44.495730 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:44.496220 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:44.496349 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:44.506880 (+ 10531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.195:59917: BlockingRecv error: recv got EOF from 127.5.213.195:59917 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 18 22:39:44 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:44 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:44.588310 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:44.579443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.193:44685)
0618 22:39:44.580175 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:44.580189 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:44.580297 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:44.580544 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:44.580546 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:44.580937 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:44.581098 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:44.581107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:44.582182 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:44.582187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:44.583227 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:44.583241 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:44.583969 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:44.583973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:44.584090 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:44.584709 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:44.584726 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:44.588053 (+ 3327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":492,"threads_started":1}
W20260618 22:39:44.588445 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:44.579552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:44685 (local address 127.5.213.254:43463)
0618 22:39:44.579741 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:44.579745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:44.580258 (+ 513us) server_negotiation.cc:407] Connection header received
0618 22:39:44.580379 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:44.580382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:44.580441 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:44.580516 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:44.581244 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:44.582029 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:44.583368 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:44.583802 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:44.583899 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:44.588235 (+ 4336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.193:44685: BlockingRecv error: recv got EOF from 127.5.213.193:44685 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 18 22:39:45 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:45 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:45.066265 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:45.057423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.194:60381)
0618 22:39:45.057918 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:45.057937 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:45.058050 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:45.058405 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:45.058411 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:45.058906 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:45.059097 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:45.059110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:45.060407 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:45.060411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:45.061345 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:45.061357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:45.062094 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:45.062099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:45.062212 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:45.062837 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:45.062854 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:45.065926 (+ 3072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":248,"threads_started":1}
W20260618 22:39:45.066375 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:45.057531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:60381 (local address 127.5.213.254:43463)
0618 22:39:45.057693 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:45.057698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:45.058013 (+ 315us) server_negotiation.cc:407] Connection header received
0618 22:39:45.058156 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:45.058161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:45.058240 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:45.058332 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:45.059249 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:45.060280 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:45.061470 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:45.061944 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:45.062035 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:45.066155 (+ 4120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.194:60381: BlockingRecv error: recv got EOF from 127.5.213.194:60381 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 18 22:39:45 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:45 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:45.516721 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:45.507995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.195:33229)
0618 22:39:45.508505 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:45.508521 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:45.508629 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:45.508930 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:45.508935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:45.509352 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:45.509523 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:45.509544 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:45.510515 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:45.510519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:45.511597 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:45.511610 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:45.512275 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:45.512280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:45.512413 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:45.513109 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:45.513129 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:45.516424 (+ 3295us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":273,"threads_started":1}
W20260618 22:39:45.516786 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:45.508089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:33229 (local address 127.5.213.254:43463)
0618 22:39:45.508270 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:45.508275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:45.508582 (+ 307us) server_negotiation.cc:407] Connection header received
0618 22:39:45.508717 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:45.508720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:45.508786 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:45.508862 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:45.509662 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:45.510374 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:45.511711 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:45.512110 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:45.512201 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:45.516592 (+ 4391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.195:33229: BlockingRecv error: recv got EOF from 127.5.213.195:33229 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 18 22:39:45 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:45 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:45.597572 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:45.589327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.193:44201)
0618 22:39:45.589880 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:45.589895 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:45.590018 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:45.590345 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:45.590350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:45.590862 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:45.591048 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:45.591060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:45.592005 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:45.592008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:45.592930 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:45.592940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:45.593551 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:45.593555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:45.593686 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:45.594200 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:45.594215 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:45.597351 (+ 3136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":320,"threads_started":1}
W20260618 22:39:45.597697 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:45.589395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:44201 (local address 127.5.213.254:43463)
0618 22:39:45.589629 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:45.589636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:45.589974 (+ 338us) server_negotiation.cc:407] Connection header received
0618 22:39:45.590141 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:45.590146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:45.590209 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:45.590288 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:45.591178 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:45.591905 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:45.593029 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:45.593404 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:45.593489 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:45.597513 (+ 4024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.193:44201: BlockingRecv error: recv got EOF from 127.5.213.193:44201 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 18 22:39:46 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:46 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:46.076974 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:46.067227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.194:45815)
0618 22:39:46.067727 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:46.067746 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:46.067867 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:46.068192 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:46.068197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:46.068586 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:46.068745 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:46.068755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:46.069980 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:46.069984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:46.070931 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:46.070943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:46.071623 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:46.071629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:46.071789 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:46.072487 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:46.072510 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:46.076651 (+ 4141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":253,"threads_started":1}
W20260618 22:39:46.077142 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:46.067328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:45815 (local address 127.5.213.254:43463)
0618 22:39:46.067489 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:46.067493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:46.067828 (+ 335us) server_negotiation.cc:407] Connection header received
0618 22:39:46.067970 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:46.067975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:46.068051 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:46.068131 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:46.068913 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:46.069856 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:46.071045 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:46.071455 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:46.071574 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:46.076892 (+ 5318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.194:45815: BlockingRecv error: recv got EOF from 127.5.213.194:45815 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 18 22:39:46 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:46 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:46.525704 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:46.517565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.195:41629)
0618 22:39:46.518004 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:46.518018 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:46.518135 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:46.518440 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:46.518445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:46.518857 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:46.518989 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:46.518998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:46.519931 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:46.519936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:46.520866 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:46.520877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:46.521619 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:46.521625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:46.521784 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:46.522253 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:46.522295 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:46.525414 (+ 3119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":236,"threads_started":1}
W20260618 22:39:46.525856 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:46.517616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:41629 (local address 127.5.213.254:43463)
0618 22:39:46.517811 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:46.517817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:46.518096 (+ 279us) server_negotiation.cc:407] Connection header received
0618 22:39:46.518249 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:46.518255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:46.518321 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:46.518399 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:46.519125 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:46.519815 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:46.520991 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:46.521485 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:46.521589 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:46.525596 (+ 4007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.195:41629: BlockingRecv error: recv got EOF from 127.5.213.195:41629 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 18 22:39:46 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:46 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:46.606568 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:46.598444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.193:47779)
0618 22:39:46.598987 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:46.599002 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:46.599133 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:46.599404 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:46.599408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:46.599767 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:46.599941 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:46.599951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:46.600896 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:46.600900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:46.601887 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:46.601899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:46.602596 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:46.602600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:46.602738 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:46.603260 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:46.603283 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:46.606333 (+ 3050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":311,"threads_started":1}
W20260618 22:39:46.606789 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:46.598505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:47779 (local address 127.5.213.254:43463)
0618 22:39:46.598727 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:46.598732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:46.599086 (+ 354us) server_negotiation.cc:407] Connection header received
0618 22:39:46.599232 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:46.599236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:46.599297 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:46.599374 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:46.600059 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:46.600799 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:46.602018 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:46.602468 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:46.602570 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:46.606501 (+ 3931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.193:47779: BlockingRecv error: recv got EOF from 127.5.213.193:47779 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 18 22:39:47 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:47 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:47.086045 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:47.078015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.194:50219)
0618 22:39:47.078439 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:47.078458 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:47.078570 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:47.078856 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:47.078859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:47.079190 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:47.079342 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:47.079350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:47.080267 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:47.080271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:47.081226 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:47.081236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:47.081870 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:47.081873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:47.081985 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:47.082573 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:47.082589 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:47.085804 (+ 3215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":225,"threads_started":1}
W20260618 22:39:47.086205 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:47.078111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:50219 (local address 127.5.213.254:43463)
0618 22:39:47.078312 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:47.078318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:47.078528 (+ 210us) server_negotiation.cc:407] Connection header received
0618 22:39:47.078679 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:47.078683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:47.078743 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:47.078811 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:47.079448 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:47.080162 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:47.081342 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:47.081732 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:47.081815 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:47.085986 (+ 4171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.194:50219: BlockingRecv error: recv got EOF from 127.5.213.194:50219 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 18 22:39:47 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:47 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:47.534571 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:47.526550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.195:60375)
0618 22:39:47.527058 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:47.527075 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:47.527180 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:47.527490 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:47.527494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:47.527852 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:47.528030 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:47.528041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:47.529084 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:47.529088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:47.530061 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:47.530072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:47.530743 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:47.530747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:47.530861 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:47.531385 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:47.531401 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:47.534346 (+ 2945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":290,"threads_started":1}
W20260618 22:39:47.534770 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:47.526694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:60375 (local address 127.5.213.254:43463)
0618 22:39:47.526882 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:47.526888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:47.527124 (+ 236us) server_negotiation.cc:407] Connection header received
0618 22:39:47.527295 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:47.527298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:47.527373 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:47.527450 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:47.528191 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:47.528961 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:47.530189 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:47.530613 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:47.530720 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:47.534516 (+ 3796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.195:60375: BlockingRecv error: recv got EOF from 127.5.213.195:60375 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 18 22:39:47 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:47 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:47.615566 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:47.607522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.193:46369)
0618 22:39:47.607987 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:47.608003 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:47.608128 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:47.608421 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:47.608426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:47.608815 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:47.608991 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:47.609000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:47.609965 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:47.609969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:47.610922 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:47.610933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:47.611598 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:47.611604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:47.611761 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:47.612320 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:47.612336 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:47.615353 (+ 3017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":271,"threads_started":1}
W20260618 22:39:47.615687 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:47.607588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:46369 (local address 127.5.213.254:43463)
0618 22:39:47.607781 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:47.607787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:47.608077 (+ 290us) server_negotiation.cc:407] Connection header received
0618 22:39:47.608239 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:47.608243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:47.608302 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:47.608376 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:47.609121 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:47.609846 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:47.611058 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:47.611459 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:47.611565 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:47.615512 (+ 3947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.193:46369: BlockingRecv error: recv got EOF from 127.5.213.193:46369 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 18 22:39:48 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:48 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:48.094934 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:48.086891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.194:42627)
0618 22:39:48.087288 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:48.087306 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:48.087414 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:48.087667 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:48.087670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:48.088064 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:48.088231 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:48.088239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:48.089219 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:48.089222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:48.090115 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:48.090125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:48.090733 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:48.090737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:48.090869 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:48.091369 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:48.091384 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:48.094620 (+ 3236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":205,"threads_started":1}
W20260618 22:39:48.095024 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:48.086974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:42627 (local address 127.5.213.254:43463)
0618 22:39:48.087178 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:48.087184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:48.087371 (+ 187us) server_negotiation.cc:407] Connection header received
0618 22:39:48.087501 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:48.087505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:48.087565 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:48.087638 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:48.088348 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:48.089114 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:48.090218 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:48.090587 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:48.090728 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:48.094804 (+ 4076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.194:42627: BlockingRecv error: recv got EOF from 127.5.213.194:42627 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 18 22:39:48 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:48 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:48.543833 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:48.535389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.195:59503)
0618 22:39:48.535852 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:48.535868 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:48.535978 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:48.536271 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:48.536275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:48.536658 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:48.536845 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:48.536853 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:48.537838 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:48.537843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:48.538790 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:48.538802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:48.539465 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:48.539469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:48.539585 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:48.540182 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:48.540199 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:48.543587 (+ 3388us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":238,"threads_started":1}
W20260618 22:39:48.543919 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:48.535500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:59503 (local address 127.5.213.254:43463)
0618 22:39:48.535660 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:48.535664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:48.535968 (+ 304us) server_negotiation.cc:407] Connection header received
0618 22:39:48.536066 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:48.536070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:48.536133 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:48.536223 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:48.536974 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:48.537711 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:48.538926 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:48.539337 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:48.539441 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:48.543750 (+ 4309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.195:59503: BlockingRecv error: recv got EOF from 127.5.213.195:59503 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 18 22:39:48 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:48 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:48.624958 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:48.616374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.193:57595)
0618 22:39:48.616823 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:48.616838 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:48.616961 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:48.617275 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:48.617280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:48.617659 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:48.617825 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:48.617835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:48.618873 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:48.618877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:48.619824 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:48.619835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:48.620572 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:48.620579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:48.620745 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:48.621281 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:48.621296 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:48.624624 (+ 3328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":226,"threads_started":1}
W20260618 22:39:48.625037 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:48.616419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:57595 (local address 127.5.213.254:43463)
0618 22:39:48.616603 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:48.616609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:48.616914 (+ 305us) server_negotiation.cc:407] Connection header received
0618 22:39:48.617065 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:48.617071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:48.617133 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:48.617212 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:48.617962 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:48.618740 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:48.619955 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:48.620396 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:48.620520 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:48.624801 (+ 4281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.193:57595: BlockingRecv error: recv got EOF from 127.5.213.193:57595 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 18 22:39:49 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:49 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:49.104383 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:49.095787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.194:48217)
0618 22:39:49.096273 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:49.096305 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:49.096430 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:49.096734 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:49.096738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:49.097143 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:49.097307 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:49.097316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:49.098319 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:49.098325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:49.099366 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:49.099379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:49.100023 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:49.100029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:49.100193 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:49.100797 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:49.100815 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:49.104130 (+ 3315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":267,"threads_started":1}
W20260618 22:39:49.104532 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:49.095863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:48217 (local address 127.5.213.254:43463)
0618 22:39:49.096041 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:49.096046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:49.096382 (+ 336us) server_negotiation.cc:407] Connection header received
0618 22:39:49.096538 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:49.096543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:49.096609 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:49.096688 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:49.097439 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:49.098183 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:49.099500 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:49.099893 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:49.099996 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:49.104329 (+ 4333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.194:48217: BlockingRecv error: recv got EOF from 127.5.213.194:48217 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 18 22:39:49 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:49 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:49.561935 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:49.544652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.195:50633)
0618 22:39:49.545202 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:49.545218 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:49.545331 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:49.545611 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:49.545615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:49.545984 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:49.546139 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:49.546147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:49.547229 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:49.547233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:49.548202 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:49.548214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:49.548914 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:49.548919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:49.549058 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:49.558265 (+ 9207us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:49.558286 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:49.561557 (+ 3271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":279,"threads_started":1}
W20260618 22:39:49.562029 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:49.544792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:50633 (local address 127.5.213.254:43463)
0618 22:39:49.544946 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:49.544950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:49.545310 (+ 360us) server_negotiation.cc:407] Connection header received
0618 22:39:49.545419 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:49.545423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:49.545497 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:49.545580 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:49.546288 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:49.547092 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:49.548335 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:49.548752 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:49.548848 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:49.561801 (+ 12953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.195:50633: BlockingRecv error: recv got EOF from 127.5.213.195:50633 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 18 22:39:49 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:49 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:49.634445 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:49.625999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.193:44575)
0618 22:39:49.626483 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:49.626501 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:49.626685 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:49.626968 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:49.626972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:49.627414 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:49.627603 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:49.627614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:49.628606 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:49.628611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:49.629593 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:49.629606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:49.630331 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:49.630336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:49.630466 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:49.631052 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:49.631069 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:49.634219 (+ 3150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":249,"threads_started":1}
W20260618 22:39:49.634619 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:49.626061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:44575 (local address 127.5.213.254:43463)
0618 22:39:49.626254 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:49.626258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:49.626595 (+ 337us) server_negotiation.cc:407] Connection header received
0618 22:39:49.626781 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:49.626784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:49.626843 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:49.626919 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:49.627722 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:49.628444 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:49.629742 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:49.630183 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:49.630277 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:49.634410 (+ 4133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.193:44575: BlockingRecv error: recv got EOF from 127.5.213.193:44575 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 18 22:39:50 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:50 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:50.113234 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:50.105182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.194:43135)
0618 22:39:50.105626 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:50.105645 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:50.105781 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:50.106079 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:50.106083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:50.106455 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:50.106673 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:50.106684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:50.107645 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:50.107650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:50.108586 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:50.108599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:50.109242 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:50.109246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:50.109366 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:50.109903 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:50.109919 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:50.113002 (+ 3083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":239,"threads_started":1}
W20260618 22:39:50.113364 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:50.105292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:43135 (local address 127.5.213.254:43463)
0618 22:39:50.105448 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:50.105452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:50.105731 (+ 279us) server_negotiation.cc:407] Connection header received
0618 22:39:50.105857 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:50.105860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:50.105917 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:50.105995 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:50.106780 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:50.107482 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:50.108696 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:50.109119 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:50.109203 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:50.113184 (+ 3981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.194:43135: BlockingRecv error: recv got EOF from 127.5.213.194:43135 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 18 22:39:50 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:50 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.195@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:50.570937 15268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:50.562955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.195:39387)
0618 22:39:50.563424 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:50.563440 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:50.563546 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:50.563805 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:50.563809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:50.564203 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:50.564344 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:50.564352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:50.565303 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:50.565307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:50.566220 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:50.566232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:50.566827 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:50.566831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:50.566950 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:50.567487 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:50.567504 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:50.570649 (+ 3145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":257,"threads_started":1}
W20260618 22:39:50.571063 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:50.563034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:39387 (local address 127.5.213.254:43463)
0618 22:39:50.563186 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:50.563190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:50.563501 (+ 311us) server_negotiation.cc:407] Connection header received
0618 22:39:50.563633 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:50.563637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:50.563696 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:50.563789 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:50.564446 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:50.565175 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:50.566333 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:50.566719 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:50.566811 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:50.570848 (+ 4037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.195:39387: BlockingRecv error: recv got EOF from 127.5.213.195:39387 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 18 22:39:50 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:50 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.193@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:50.644021 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:50.635437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.193:49197)
0618 22:39:50.635932 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:50.635946 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:50.636066 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:50.636404 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:50.636409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:50.636801 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:50.636977 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:50.636986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:50.637922 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:50.637928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:50.638921 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:50.638933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:50.639650 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:50.639655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:50.639837 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:50.640447 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:50.640469 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:50.643683 (+ 3214us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":252,"threads_started":1}
W20260618 22:39:50.644104 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:50.635510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:49197 (local address 127.5.213.254:43463)
0618 22:39:50.635774 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:50.635780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:50.636018 (+ 238us) server_negotiation.cc:407] Connection header received
0618 22:39:50.636183 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:50.636186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:50.636253 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:50.636352 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:50.637110 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:50.637787 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:50.639063 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:50.639521 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:50.639624 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:50.643879 (+ 4255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.193:49197: BlockingRecv error: recv got EOF from 127.5.213.193:49197 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Jun 18 22:39:51 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:39:51 dist-test-slave-lnj3 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.213.194@KRBTEST.COM for oryx/127.5.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:39:51.123063 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:51.114318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43463 (local address 127.5.213.194:54167)
0618 22:39:51.114801 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:51.114820 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:51.114941 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:51.115232 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:51.115236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:51.115596 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:51.115786 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:51.115796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:51.116796 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:51.116800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:51.117845 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:51.117859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:51.118693 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:51.118700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:51.118843 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:51.119428 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:51.119447 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:51.122747 (+ 3300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.213.254:43463: Server oryx/127.5.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":232,"threads_started":1}
W20260618 22:39:51.123180 15226 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:51.114399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:54167 (local address 127.5.213.254:43463)
0618 22:39:51.114580 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:51.114587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:51.114892 (+ 305us) server_negotiation.cc:407] Connection header received
0618 22:39:51.115054 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:51.115058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:51.115115 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:51.115190 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:51.115918 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:51.116649 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:51.117999 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:51.118460 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:51.118571 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:51.122962 (+ 4391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.213.194:54167: BlockingRecv error: recv got EOF from 127.5.213.194:54167 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:39:51.346032 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 14810
I20260618 22:39:51.357681 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:51.347002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49452 (local address 127.5.213.193:43995)
0618 22:39:51.347549 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:51.347557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:51.347584 (+ 27us) server_negotiation.cc:407] Connection header received
0618 22:39:51.347644 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:51.347650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:51.347877 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:51.348037 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:51.349288 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:51.350311 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:51.351481 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:51.352006 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:51.352099 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:51.354762 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:51.354771 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:51.354789 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:51.354827 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:51.357167 (+ 2340us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.213.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0618 22:39:51.357454 (+ 287us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49452: Request ticket server kudu/127.5.213.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":240,"threads_started":1}
W20260618 22:39:51.357776 15272 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49452: Request ticket server kudu/127.5.213.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260618 22:39:51.357867 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 14810: Not authorized: Client connection negotiation failed: client connection to 127.5.213.193:43995: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.213.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260618 22:39:51.358062 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 14810
I20260618 22:39:51.371475 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 14961
I20260618 22:39:51.383481 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:51.372513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42596 (local address 127.5.213.194:41549)
0618 22:39:51.373114 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:51.373122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:51.373149 (+ 27us) server_negotiation.cc:407] Connection header received
0618 22:39:51.373211 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:51.373218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:51.373438 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:51.373618 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:51.374921 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:51.375956 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:51.377161 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:51.377712 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:51.377828 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:51.380239 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:51.380248 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:51.380268 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:51.380309 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:51.382822 (+ 2513us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.213.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0618 22:39:51.383172 (+ 350us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42596: Request ticket server kudu/127.5.213.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":295,"threads_started":1}
W20260618 22:39:51.383497 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 14961: Not authorized: Client connection negotiation failed: client connection to 127.5.213.194:41549: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.213.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260618 22:39:51.383582 15273 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42596: Request ticket server kudu/127.5.213.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260618 22:39:51.383623 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 14961
I20260618 22:39:51.396456 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 15094
I20260618 22:39:51.408329 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:51.397447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51598 (local address 127.5.213.195:43741)
0618 22:39:51.397942 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:51.397949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:51.397974 (+ 25us) server_negotiation.cc:407] Connection header received
0618 22:39:51.398035 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:51.398041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:51.398331 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:51.398493 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:51.400403 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:51.401401 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:51.402508 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:51.403099 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:51.403194 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:51.405414 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:51.405423 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:51.405435 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:51.405475 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:51.407816 (+ 2341us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.213.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0618 22:39:51.408075 (+ 259us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51598: Request ticket server kudu/127.5.213.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":218,"threads_started":1}
W20260618 22:39:51.408432 15274 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51598: Request ticket server kudu/127.5.213.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260618 22:39:51.408409 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 15094: Not authorized: Client connection negotiation failed: client connection to 127.5.213.195:43741: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.213.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260618 22:39:51.408568 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 15094
I20260618 22:39:51.421025 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 14741
I20260618 22:39:51.422248 14803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:51.533262 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 14741
2026-06-18T22:39:51Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20853 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-18T22:39:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:39:51Z Disabled control of system clock
I20260618 22:39:51.600455 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:43371
--webserver_interface=127.5.213.254
--webserver_port=0
--builtin_ntp_servers=127.5.213.212:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:43371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260618 22:39:51.910976 15279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:51.911255 15279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:51.911332 15279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:51.918416 15279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:39:51.918514 15279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:51.918548 15279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:51.918610 15279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:39:51.918669 15279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:39:51.928180 15279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:43371
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:43371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:51.929423 15279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:51.931173 15279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:51.942392 15284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:51.942349 15287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:51.942317 15285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:51.943085 15279 server_base.cc:1034] running on GCE node
I20260618 22:39:51.943799 15279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:51.945116 15279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:51.946308 15279 hybrid_clock.cc:648] HybridClock initialized: now 1781822391946259 us; error 30 us; skew 500 ppm
I20260618 22:39:51.946600 15279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:39:51.948801 15279 webserver.cc:460] Webserver started at http://127.5.213.254:43467/ using document root <none> and password file <none>
I20260618 22:39:51.949452 15279 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:51.949542 15279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:51.949826 15279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:51.952606 15279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "055e8b79fff24696a341a9343cc74fc7"
format_stamp: "Formatted at 2026-06-18 22:39:51 on dist-test-slave-lnj3"
server_key: "29b2803dc535c3719c55c33c285853c5"
server_key_iv: "5a06b0e250f4308a6ed2595091f32b51"
server_key_version: "encryptionkey@0"
I20260618 22:39:51.953328 15279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "055e8b79fff24696a341a9343cc74fc7"
format_stamp: "Formatted at 2026-06-18 22:39:51 on dist-test-slave-lnj3"
server_key: "29b2803dc535c3719c55c33c285853c5"
server_key_iv: "5a06b0e250f4308a6ed2595091f32b51"
server_key_version: "encryptionkey@0"
I20260618 22:39:51.958287 15279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260618 22:39:51.961495 15293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:51.962755 15279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:39:51.962937 15279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "055e8b79fff24696a341a9343cc74fc7"
format_stamp: "Formatted at 2026-06-18 22:39:51 on dist-test-slave-lnj3"
server_key: "29b2803dc535c3719c55c33c285853c5"
server_key_iv: "5a06b0e250f4308a6ed2595091f32b51"
server_key_version: "encryptionkey@0"
I20260618 22:39:51.963130 15279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:51.987131 15279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:51.988178 15279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:51.988451 15279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:52.005127 15279 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:43371
I20260618 22:39:52.005187 15344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:43371 every 8 connection(s)
I20260618 22:39:52.006428 15279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:39:52.008787 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 15279
I20260618 22:39:52.008966 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:39:52.009555 5975 external_mini_cluster.cc:1428] Setting key 0398aa17ef1fe95bb67fe916027279ef
I20260618 22:39:52.010185 15345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:52.023243 15345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7: Bootstrap starting.
I20260618 22:39:52.023608 15348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:52.011389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.5.213.254:43371)
0618 22:39:52.012277 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:52.012295 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:52.012335 (+ 40us) server_negotiation.cc:407] Connection header received
0618 22:39:52.014527 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:52.014549 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:52.015051 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:52.015419 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:52.016107 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:52.017418 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:52.019241 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:52.019939 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:52.020076 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:52.021286 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:52.021293 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:52.021307 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:52.021846 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:52.021919 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:52.021939 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:52.022031 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:52.022249 (+ 218us) server_negotiation.cc:299] Negotiation successful
0618 22:39:52.022565 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":295,"threads_started":1}
I20260618 22:39:52.026594 15345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:52.027616 15345 log.cc:824] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:52.030205 15345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7: No bootstrap required, opened a new log
I20260618 22:39:52.037331 15345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055e8b79fff24696a341a9343cc74fc7" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43371 } }
I20260618 22:39:52.037688 15345 raft_consensus.cc:374] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:52.037792 15345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055e8b79fff24696a341a9343cc74fc7, State: Initialized, Role: FOLLOWER
I20260618 22:39:52.038410 15345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055e8b79fff24696a341a9343cc74fc7" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43371 } }
I20260618 22:39:52.038700 15345 raft_consensus.cc:388] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:39:52.038864 15345 raft_consensus.cc:482] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:39:52.039032 15345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:52.041065 15345 raft_consensus.cc:504] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055e8b79fff24696a341a9343cc74fc7" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43371 } }
I20260618 22:39:52.041534 15345 leader_election.cc:302] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 055e8b79fff24696a341a9343cc74fc7; no voters:
I20260618 22:39:52.042058 15345 leader_election.cc:288] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:39:52.042193 15350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:52.042807 15350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7 [term 1 LEADER]: Becoming Leader. State: Replica: 055e8b79fff24696a341a9343cc74fc7, State: Running, Role: LEADER
I20260618 22:39:52.043301 15350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055e8b79fff24696a341a9343cc74fc7" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43371 } }
I20260618 22:39:52.043959 15345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:39:52.046038 15352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 055e8b79fff24696a341a9343cc74fc7. Latest consensus state: current_term: 1 leader_uuid: "055e8b79fff24696a341a9343cc74fc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055e8b79fff24696a341a9343cc74fc7" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43371 } } }
I20260618 22:39:52.046165 15351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "055e8b79fff24696a341a9343cc74fc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055e8b79fff24696a341a9343cc74fc7" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43371 } } }
I20260618 22:39:52.046509 15351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:52.047137 15352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:52.051779 15356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:39:52.055943 15356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:39:52.063535 15356 catalog_manager.cc:1347] Generated new cluster ID: d7663cb4b45644f4871b0cdf78be02e3
I20260618 22:39:52.063649 15356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:39:52.075155 15356 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:39:52.076398 15356 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:39:52.083534 15356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 055e8b79fff24696a341a9343cc74fc7: Generated new TSK 0
I20260618 22:39:52.084283 15356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:39:52.097752 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:43371
--builtin_ntp_servers=127.5.213.212:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260618 22:39:52.405012 15369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:52.405288 15369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:52.405395 15369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:52.412360 15369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:52.412459 15369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:52.412617 15369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:39:52.421363 15369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:43371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:52.422585 15369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:52.424270 15369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:52.436522 15377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:52.436561 15375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:52.437054 15374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:52.437335 15369 server_base.cc:1034] running on GCE node
I20260618 22:39:52.437919 15369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:52.438830 15369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:52.440019 15369 hybrid_clock.cc:648] HybridClock initialized: now 1781822392439962 us; error 53 us; skew 500 ppm
I20260618 22:39:52.440308 15369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:39:52.442364 15369 webserver.cc:460] Webserver started at http://127.5.213.193:41033/ using document root <none> and password file <none>
I20260618 22:39:52.443037 15369 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:52.443133 15369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:52.443431 15369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:52.446194 15369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "ac5eee6858944c15b5bc85d6c191bff6"
format_stamp: "Formatted at 2026-06-18 22:39:52 on dist-test-slave-lnj3"
server_key: "b73af18187ee10c63bddebc033a2224f"
server_key_iv: "0e8b28ffbef3702cb359c2e11b80963b"
server_key_version: "encryptionkey@0"
I20260618 22:39:52.446990 15369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "ac5eee6858944c15b5bc85d6c191bff6"
format_stamp: "Formatted at 2026-06-18 22:39:52 on dist-test-slave-lnj3"
server_key: "b73af18187ee10c63bddebc033a2224f"
server_key_iv: "0e8b28ffbef3702cb359c2e11b80963b"
server_key_version: "encryptionkey@0"
I20260618 22:39:52.451889 15369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260618 22:39:52.455050 15383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:52.456292 15369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260618 22:39:52.456430 15369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "ac5eee6858944c15b5bc85d6c191bff6"
format_stamp: "Formatted at 2026-06-18 22:39:52 on dist-test-slave-lnj3"
server_key: "b73af18187ee10c63bddebc033a2224f"
server_key_iv: "0e8b28ffbef3702cb359c2e11b80963b"
server_key_version: "encryptionkey@0"
I20260618 22:39:52.456610 15369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:52.477361 15369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:52.478274 15369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:52.478538 15369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:52.479473 15369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:52.480809 15369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:52.480893 15369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:52.480968 15369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:52.481020 15369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:52.510095 15369 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:46085
I20260618 22:39:52.510178 15495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:46085 every 8 connection(s)
I20260618 22:39:52.511375 15369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:39:52.517324 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 15369
I20260618 22:39:52.517495 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:39:52.518078 5975 external_mini_cluster.cc:1428] Setting key 9d10dbabadc43aec11f7c1ea19880865
I20260618 22:39:52.520900 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:43371
--builtin_ntp_servers=127.5.213.212:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260618 22:39:52.524194 15348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:52.513926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:34871 (local address 127.5.213.254:43371)
0618 22:39:52.514091 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:52.514095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:52.515466 (+ 1371us) server_negotiation.cc:407] Connection header received
0618 22:39:52.517172 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:52.517177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:52.517257 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:52.517340 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:52.518406 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:52.519328 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:52.521525 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:52.521932 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:52.522025 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:52.523524 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:52.523530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:52.523534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:52.523552 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:52.523586 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:52.523599 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:52.523685 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:52.523862 (+ 177us) server_negotiation.cc:299] Negotiation successful
0618 22:39:52.523938 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:39:52.525411 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:52.514419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43371 (local address 127.5.213.193:34871)
0618 22:39:52.515328 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:52.515364 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:52.516922 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:52.517570 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:52.517582 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:52.517684 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:52.518262 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:52.518280 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:52.519492 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:52.519496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:52.521375 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:52.521389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:52.522136 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:52.522143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:52.522436 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:52.523187 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:39:52.523190 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:52.523243 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:39:52.523343 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:52.523681 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:52.523689 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:52.523695 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 22:39:52.523856 (+ 161us) client_negotiation.cc:240] Negotiation successful
0618 22:39:52.524215 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":282,"threads_started":1}
I20260618 22:39:52.526769 15496 heartbeater.cc:347] Connected to a master server at 127.5.213.254:43371
I20260618 22:39:52.527086 15496 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:52.528028 15496 heartbeater.cc:511] Master 127.5.213.254:43371 requested a full tablet report, sending...
I20260618 22:39:52.530201 15310 ts_manager.cc:194] Registered new tserver with Master: ac5eee6858944c15b5bc85d6c191bff6 (127.5.213.193:46085)
I20260618 22:39:52.531656 15310 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.213.193:34871
W20260618 22:39:52.832497 15499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:52.832762 15499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:52.832842 15499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:52.839823 15499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:52.839915 15499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:52.840016 15499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:39:52.848755 15499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:43371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:52.849886 15499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:52.851528 15499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:52.863312 15507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:52.863360 15505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:52.863386 15504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:52.864099 15499 server_base.cc:1034] running on GCE node
I20260618 22:39:52.864581 15499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:52.865388 15499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:52.866580 15499 hybrid_clock.cc:648] HybridClock initialized: now 1781822392866525 us; error 47 us; skew 500 ppm
I20260618 22:39:52.866906 15499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:39:52.868952 15499 webserver.cc:460] Webserver started at http://127.5.213.194:44601/ using document root <none> and password file <none>
I20260618 22:39:52.869583 15499 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:52.869671 15499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:52.869958 15499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:52.872565 15499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "0e3eba066c924c8ba63caf8610360edb"
format_stamp: "Formatted at 2026-06-18 22:39:52 on dist-test-slave-lnj3"
server_key: "310c4f1266766a35d9266b406a3a4cd5"
server_key_iv: "4dd14c68f6f54f1fdfa6b90c5a099adb"
server_key_version: "encryptionkey@0"
I20260618 22:39:52.873320 15499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "0e3eba066c924c8ba63caf8610360edb"
format_stamp: "Formatted at 2026-06-18 22:39:52 on dist-test-slave-lnj3"
server_key: "310c4f1266766a35d9266b406a3a4cd5"
server_key_iv: "4dd14c68f6f54f1fdfa6b90c5a099adb"
server_key_version: "encryptionkey@0"
I20260618 22:39:52.878002 15499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260618 22:39:52.881184 15513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:52.882462 15499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:39:52.882598 15499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "0e3eba066c924c8ba63caf8610360edb"
format_stamp: "Formatted at 2026-06-18 22:39:52 on dist-test-slave-lnj3"
server_key: "310c4f1266766a35d9266b406a3a4cd5"
server_key_iv: "4dd14c68f6f54f1fdfa6b90c5a099adb"
server_key_version: "encryptionkey@0"
I20260618 22:39:52.882822 15499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:52.894337 15499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:52.895152 15499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:52.895427 15499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:52.896317 15499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:52.897660 15499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:52.897746 15499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:52.897850 15499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:52.897919 15499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:52.927868 15499 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:40983
I20260618 22:39:52.927919 15625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:40983 every 8 connection(s)
I20260618 22:39:52.929196 15499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:39:52.931922 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 15499
I20260618 22:39:52.932081 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:39:52.932598 5975 external_mini_cluster.cc:1428] Setting key 1b2665384c5c401ff30c416a401066ff
I20260618 22:39:52.935437 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:43371
--builtin_ntp_servers=127.5.213.212:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260618 22:39:52.942288 15348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:52.931740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:38219 (local address 127.5.213.254:43371)
0618 22:39:52.931896 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:52.931900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:52.933729 (+ 1829us) server_negotiation.cc:407] Connection header received
0618 22:39:52.935771 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:52.935776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:52.935866 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:52.935960 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:52.936999 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:52.937953 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:52.939585 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:52.940166 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:52.940289 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:52.941528 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:52.941534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:52.941538 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:52.941561 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:52.941605 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:52.941620 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:52.941713 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:52.941900 (+ 187us) server_negotiation.cc:299] Negotiation successful
0618 22:39:52.941999 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:39:52.943372 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:52.932516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43371 (local address 127.5.213.194:38219)
0618 22:39:52.933579 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:52.933623 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:52.935479 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:52.936151 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:52.936170 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:52.936270 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:52.936846 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:52.936863 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:52.938076 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:52.938079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:52.939443 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:52.939454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:52.940364 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:52.940369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:52.940568 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:52.941227 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:39:52.941229 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:52.941274 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:39:52.941353 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:52.941704 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:52.941710 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:52.941715 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:39:52.941871 (+ 156us) client_negotiation.cc:240] Negotiation successful
0618 22:39:52.942289 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":297,"threads_started":1}
I20260618 22:39:52.944780 15626 heartbeater.cc:347] Connected to a master server at 127.5.213.254:43371
I20260618 22:39:52.945050 15626 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:52.945802 15626 heartbeater.cc:511] Master 127.5.213.254:43371 requested a full tablet report, sending...
I20260618 22:39:52.947185 15310 ts_manager.cc:194] Registered new tserver with Master: 0e3eba066c924c8ba63caf8610360edb (127.5.213.194:40983)
I20260618 22:39:52.948164 15310 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.213.194:38219
W20260618 22:39:53.247848 15629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:53.248117 15629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:53.248243 15629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:53.255489 15629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:53.255592 15629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:53.255739 15629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:39:53.265163 15629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:43371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:53.266492 15629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:53.268374 15629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:53.280659 15637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:53.280737 15634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:53.280715 15635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:53.281756 15629 server_base.cc:1034] running on GCE node
I20260618 22:39:53.282370 15629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:53.283339 15629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:53.284538 15629 hybrid_clock.cc:648] HybridClock initialized: now 1781822393284478 us; error 56 us; skew 500 ppm
I20260618 22:39:53.284863 15629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:39:53.287128 15629 webserver.cc:460] Webserver started at http://127.5.213.195:35217/ using document root <none> and password file <none>
I20260618 22:39:53.287851 15629 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:53.287954 15629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:53.288259 15629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:53.291226 15629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "750469874fbe456a8f3e9b4050af221d"
format_stamp: "Formatted at 2026-06-18 22:39:53 on dist-test-slave-lnj3"
server_key: "da7402817ef3f56bd1b94976804ccf56"
server_key_iv: "f7be7d14a0c40c9246852e49576f1ea6"
server_key_version: "encryptionkey@0"
I20260618 22:39:53.292042 15629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "750469874fbe456a8f3e9b4050af221d"
format_stamp: "Formatted at 2026-06-18 22:39:53 on dist-test-slave-lnj3"
server_key: "da7402817ef3f56bd1b94976804ccf56"
server_key_iv: "f7be7d14a0c40c9246852e49576f1ea6"
server_key_version: "encryptionkey@0"
I20260618 22:39:53.297214 15629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:39:53.301211 15643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:53.302783 15629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260618 22:39:53.302965 15629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "750469874fbe456a8f3e9b4050af221d"
format_stamp: "Formatted at 2026-06-18 22:39:53 on dist-test-slave-lnj3"
server_key: "da7402817ef3f56bd1b94976804ccf56"
server_key_iv: "f7be7d14a0c40c9246852e49576f1ea6"
server_key_version: "encryptionkey@0"
I20260618 22:39:53.303179 15629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:53.322834 15629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:53.323707 15629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:53.323997 15629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:53.325001 15629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:53.326463 15629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:53.326579 15629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:53.326687 15629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:53.326743 15629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:53.356812 15629 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:43935
I20260618 22:39:53.356848 15755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:43935 every 8 connection(s)
I20260618 22:39:53.358191 15629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:39:53.367625 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 15629
I20260618 22:39:53.367792 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:39:53.368257 5975 external_mini_cluster.cc:1428] Setting key f05e28ab54d9df41fb93635caa66e57c
I20260618 22:39:53.372026 15348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:53.360918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:59061 (local address 127.5.213.254:43371)
0618 22:39:53.361110 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:53.361116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:53.362827 (+ 1711us) server_negotiation.cc:407] Connection header received
0618 22:39:53.364859 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:53.364864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:53.364936 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:53.365041 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:53.366312 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:53.367144 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:53.369295 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:53.369836 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:53.369970 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:53.371420 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:53.371426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:53.371429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:53.371447 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:53.371478 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:53.371490 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:53.371561 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:53.371726 (+ 165us) server_negotiation.cc:299] Negotiation successful
0618 22:39:53.371801 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:39:53.373064 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:53.361542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43371 (local address 127.5.213.195:59061)
0618 22:39:53.362611 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:53.362706 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:53.364452 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:53.365312 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:53.365322 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:53.365426 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:53.366157 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:53.366180 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:53.367325 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:53.367331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:53.368892 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:53.368908 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:53.370054 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:53.370061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:53.370322 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:53.371014 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:39:53.371017 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:53.371075 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:39:53.371188 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:53.371566 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:53.371573 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:53.371578 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:39:53.371713 (+ 135us) client_negotiation.cc:240] Negotiation successful
0618 22:39:53.372027 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":313,"threads_started":1}
I20260618 22:39:53.374464 15756 heartbeater.cc:347] Connected to a master server at 127.5.213.254:43371
I20260618 22:39:53.375012 15756 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:53.375819 15756 heartbeater.cc:511] Master 127.5.213.254:43371 requested a full tablet report, sending...
I20260618 22:39:53.377136 15310 ts_manager.cc:194] Registered new tserver with Master: 750469874fbe456a8f3e9b4050af221d (127.5.213.195:43935)
I20260618 22:39:53.377862 15310 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.213.195:59061
I20260618 22:39:53.383459 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260618 22:39:53.389423 15348 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:53.387982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40036 (local address 127.5.213.254:43371)
0618 22:39:53.388132 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:53.388136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:53.388534 (+ 398us) server_negotiation.cc:407] Connection header received
0618 22:39:53.388751 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:53.388756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:53.388829 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:53.388908 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:53.389223 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40036: BlockingRecv error: recv got EOF from 127.0.0.1:40036 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:39:53.390127 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 15369
I20260618 22:39:53.397744 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:53.391019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44414 (local address 127.5.213.193:46085)
0618 22:39:53.391540 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:53.391547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:53.391574 (+ 27us) server_negotiation.cc:407] Connection header received
0618 22:39:53.391663 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:53.391669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:53.391921 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:53.392051 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:53.392457 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:53.393477 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:53.394771 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:53.395417 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:53.395511 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:53.396457 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:53.396464 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:53.396477 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:53.397053 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:53.397096 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:53.397115 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:53.397203 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:53.397360 (+ 157us) server_negotiation.cc:299] Negotiation successful
0618 22:39:53.397465 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":254,"threads_started":1}
I20260618 22:39:53.398994 15491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:53.478775 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 15369
I20260618 22:39:53.491677 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 15499
I20260618 22:39:53.499329 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:53.492559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.5.213.194:40983)
0618 22:39:53.493051 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:53.493058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:53.493083 (+ 25us) server_negotiation.cc:407] Connection header received
0618 22:39:53.493142 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:53.493148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:53.493373 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:53.493520 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:53.493950 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:53.494991 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:53.496214 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:53.496792 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:53.496903 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:53.497902 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:53.497910 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:53.497924 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:53.498513 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:53.498560 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:53.498580 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:53.498726 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:53.498859 (+ 133us) server_negotiation.cc:299] Negotiation successful
0618 22:39:53.499007 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":229,"threads_started":1}
I20260618 22:39:53.500545 15621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:53.580379 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 15499
I20260618 22:39:53.593343 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 15629
I20260618 22:39:53.601233 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:53.594357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35854 (local address 127.5.213.195:43935)
0618 22:39:53.595007 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:53.595014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:53.595044 (+ 30us) server_negotiation.cc:407] Connection header received
0618 22:39:53.595101 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:53.595107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:53.595324 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:53.595451 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:53.595834 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:53.596883 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:53.598083 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:53.598702 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:53.598798 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:53.599840 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:53.599849 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:53.599862 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:53.600421 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:53.600467 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:53.600485 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:53.600571 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:53.600732 (+ 161us) server_negotiation.cc:299] Negotiation successful
0618 22:39:53.600853 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":354,"threads_started":1}
I20260618 22:39:53.602504 15751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:53.682986 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 15629
I20260618 22:39:53.695163 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 15279
I20260618 22:39:53.696312 15340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:53.795738 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 15279
2026-06-18T22:39:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2257 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-18T22:39:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:39:53Z Disabled control of system clock
I20260618 22:39:53.859804 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:33519
--webserver_interface=127.5.213.254
--webserver_port=0
--builtin_ntp_servers=127.5.213.212:44525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:33519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260618 22:39:54.174870 15775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:54.175160 15775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:54.175280 15775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:54.182461 15775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:39:54.182571 15775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:54.182647 15775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:54.182729 15775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:39:54.182778 15775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:39:54.191911 15775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:44525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:33519
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:33519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:54.193133 15775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:54.194878 15775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:54.206218 15781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:54.206447 15780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:54.206187 15783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:54.207346 15775 server_base.cc:1034] running on GCE node
I20260618 22:39:54.208035 15775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:54.209450 15775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:54.210703 15775 hybrid_clock.cc:648] HybridClock initialized: now 1781822394210657 us; error 31 us; skew 500 ppm
I20260618 22:39:54.211018 15775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:39:54.213130 15775 webserver.cc:460] Webserver started at http://127.5.213.254:42373/ using document root <none> and password file <none>
I20260618 22:39:54.213773 15775 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:54.213855 15775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:54.214118 15775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:54.216949 15775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "b338d78f5a944a5486f5daececefb38e"
format_stamp: "Formatted at 2026-06-18 22:39:54 on dist-test-slave-lnj3"
server_key: "64ed0b7de50cd85f5d4dedd2a6a92be4"
server_key_iv: "f8f53f8fcdec3bfe803921fdda17a841"
server_key_version: "encryptionkey@0"
I20260618 22:39:54.217669 15775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "b338d78f5a944a5486f5daececefb38e"
format_stamp: "Formatted at 2026-06-18 22:39:54 on dist-test-slave-lnj3"
server_key: "64ed0b7de50cd85f5d4dedd2a6a92be4"
server_key_iv: "f8f53f8fcdec3bfe803921fdda17a841"
server_key_version: "encryptionkey@0"
I20260618 22:39:54.222456 15775 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260618 22:39:54.225764 15789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:54.227227 15775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:39:54.227366 15775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "b338d78f5a944a5486f5daececefb38e"
format_stamp: "Formatted at 2026-06-18 22:39:54 on dist-test-slave-lnj3"
server_key: "64ed0b7de50cd85f5d4dedd2a6a92be4"
server_key_iv: "f8f53f8fcdec3bfe803921fdda17a841"
server_key_version: "encryptionkey@0"
I20260618 22:39:54.227538 15775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:54.232734 15775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:54.233382 15775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:54.233614 15775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:54.249653 15775 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:33519
I20260618 22:39:54.249683 15840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:33519 every 8 connection(s)
I20260618 22:39:54.250916 15775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:39:54.254375 15841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:54.259342 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 15775
I20260618 22:39:54.259544 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:39:54.260174 5975 external_mini_cluster.cc:1428] Setting key 4ec72157cf26f2757767c7f88c8301ce
I20260618 22:39:54.264724 15841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e: Bootstrap starting.
I20260618 22:39:54.267887 15841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:54.268127 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:54.262082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60058 (local address 127.5.213.254:33519)
0618 22:39:54.263109 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:54.263120 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:54.263159 (+ 39us) server_negotiation.cc:407] Connection header received
0618 22:39:54.264684 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:54.264705 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:54.265183 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:54.265518 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:54.266529 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:54.266535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:54.266547 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:54.266571 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:54.266657 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:54.266685 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:54.266870 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:54.266994 (+ 124us) server_negotiation.cc:299] Negotiation successful
0618 22:39:54.267233 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":321,"threads_started":1}
I20260618 22:39:54.268958 15841 log.cc:824] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:54.271734 15841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e: No bootstrap required, opened a new log
I20260618 22:39:54.279132 15841 raft_consensus.cc:348] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b338d78f5a944a5486f5daececefb38e" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33519 } }
I20260618 22:39:54.279505 15841 raft_consensus.cc:374] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:54.279603 15841 raft_consensus.cc:729] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b338d78f5a944a5486f5daececefb38e, State: Initialized, Role: FOLLOWER
I20260618 22:39:54.280114 15841 consensus_queue.cc:260] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b338d78f5a944a5486f5daececefb38e" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33519 } }
I20260618 22:39:54.280319 15841 raft_consensus.cc:388] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:39:54.280447 15841 raft_consensus.cc:482] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:39:54.280602 15841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:54.283120 15841 raft_consensus.cc:504] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b338d78f5a944a5486f5daececefb38e" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33519 } }
I20260618 22:39:54.283612 15841 leader_election.cc:302] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b338d78f5a944a5486f5daececefb38e; no voters:
I20260618 22:39:54.284188 15841 leader_election.cc:288] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:39:54.284296 15846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:54.284907 15846 raft_consensus.cc:686] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e [term 1 LEADER]: Becoming Leader. State: Replica: b338d78f5a944a5486f5daececefb38e, State: Running, Role: LEADER
I20260618 22:39:54.285468 15846 consensus_queue.cc:237] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b338d78f5a944a5486f5daececefb38e" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33519 } }
I20260618 22:39:54.286100 15841 sys_catalog.cc:564] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:39:54.288060 15848 sys_catalog.cc:455] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b338d78f5a944a5486f5daececefb38e. Latest consensus state: current_term: 1 leader_uuid: "b338d78f5a944a5486f5daececefb38e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b338d78f5a944a5486f5daececefb38e" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33519 } } }
I20260618 22:39:54.288435 15848 sys_catalog.cc:458] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:54.288221 15847 sys_catalog.cc:455] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b338d78f5a944a5486f5daececefb38e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b338d78f5a944a5486f5daececefb38e" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 33519 } } }
I20260618 22:39:54.288525 15847 sys_catalog.cc:458] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:54.291189 15854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:39:54.296036 15854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:39:54.305107 15854 catalog_manager.cc:1347] Generated new cluster ID: a2f80e503f7c478c96c7856c6ce71005
I20260618 22:39:54.305220 15854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:39:54.324261 15854 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:39:54.324729 15854 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:39:54.334805 15854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b338d78f5a944a5486f5daececefb38e: Generated new TSK 0
I20260618 22:39:54.335904 15854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:39:54.342759 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:33519
--builtin_ntp_servers=127.5.213.212:44525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260618 22:39:54.653061 15865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:54.653333 15865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:54.653468 15865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:54.660601 15865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:54.660712 15865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:54.660858 15865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:39:54.669756 15865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:44525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:33519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:54.671054 15865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:54.672777 15865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:54.685045 15870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:54.685068 15871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:54.685320 15873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:54.685756 15865 server_base.cc:1034] running on GCE node
I20260618 22:39:54.686285 15865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:54.687251 15865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:54.688453 15865 hybrid_clock.cc:648] HybridClock initialized: now 1781822394688401 us; error 47 us; skew 500 ppm
I20260618 22:39:54.688771 15865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:39:54.694294 15865 webserver.cc:460] Webserver started at http://127.5.213.193:35529/ using document root <none> and password file <none>
I20260618 22:39:54.694989 15865 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:54.695075 15865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:54.695341 15865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:54.697990 15865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "9972771118f744e8bcaa4067443dcdea"
format_stamp: "Formatted at 2026-06-18 22:39:54 on dist-test-slave-lnj3"
server_key: "776e9edb31c007efb661174a3bf71789"
server_key_iv: "a3c3f4214d1e823a63478334945f8cd6"
server_key_version: "encryptionkey@0"
I20260618 22:39:54.698770 15865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "9972771118f744e8bcaa4067443dcdea"
format_stamp: "Formatted at 2026-06-18 22:39:54 on dist-test-slave-lnj3"
server_key: "776e9edb31c007efb661174a3bf71789"
server_key_iv: "a3c3f4214d1e823a63478334945f8cd6"
server_key_version: "encryptionkey@0"
I20260618 22:39:54.703532 15865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260618 22:39:54.706746 15879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:54.707974 15865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:39:54.708114 15865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "9972771118f744e8bcaa4067443dcdea"
format_stamp: "Formatted at 2026-06-18 22:39:54 on dist-test-slave-lnj3"
server_key: "776e9edb31c007efb661174a3bf71789"
server_key_iv: "a3c3f4214d1e823a63478334945f8cd6"
server_key_version: "encryptionkey@0"
I20260618 22:39:54.708294 15865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:54.713557 15865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:54.714251 15865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:54.714479 15865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:54.715339 15865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:54.716622 15865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:54.716758 15865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:54.716876 15865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:54.716972 15865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:54.746345 15865 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:38255
I20260618 22:39:54.746421 15991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:38255 every 8 connection(s)
I20260618 22:39:54.747639 15865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:39:54.752804 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 15865
I20260618 22:39:54.752983 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:39:54.753688 5975 external_mini_cluster.cc:1428] Setting key 5d44b4f11bea2dc59c4b3d6011dd3da3
I20260618 22:39:54.755254 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:54.749999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:45079 (local address 127.5.213.254:33519)
0618 22:39:54.750153 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:54.750157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:54.751502 (+ 1345us) server_negotiation.cc:407] Connection header received
0618 22:39:54.753150 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:54.753156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:54.753222 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:54.753327 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:54.754544 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:54.754548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:54.754551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:54.754567 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:54.754596 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:54.754608 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:54.754775 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:54.754946 (+ 171us) server_negotiation.cc:299] Negotiation successful
0618 22:39:54.755007 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:39:54.756209 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:33519
--builtin_ntp_servers=127.5.213.212:44525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260618 22:39:54.756223 15994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:54.750472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33519 (local address 127.5.213.193:45079)
0618 22:39:54.751369 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:54.751410 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:54.752839 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:54.753507 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:54.753515 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:54.753608 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:54.754232 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:39:54.754235 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:54.754271 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:39:54.754380 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:54.754743 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:54.754754 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:54.754762 (+ 8us) client_negotiation.cc:769] Sending connection context
0618 22:39:54.755019 (+ 257us) client_negotiation.cc:240] Negotiation successful
0618 22:39:54.755282 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":280,"threads_started":1}
I20260618 22:39:54.757635 15992 heartbeater.cc:347] Connected to a master server at 127.5.213.254:33519
I20260618 22:39:54.757918 15992 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:54.758430 15992 heartbeater.cc:511] Master 127.5.213.254:33519 requested a full tablet report, sending...
I20260618 22:39:54.760490 15806 ts_manager.cc:194] Registered new tserver with Master: 9972771118f744e8bcaa4067443dcdea (127.5.213.193:38255)
W20260618 22:39:55.076095 15995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:55.076411 15995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:55.076500 15995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:55.083683 15995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:55.083782 15995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:55.083899 15995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:39:55.093112 15995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:44525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:33519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:55.094367 15995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:55.096112 15995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:55.109009 16000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:55.109054 16003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:55.108985 16001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:55.109997 15995 server_base.cc:1034] running on GCE node
I20260618 22:39:55.110602 15995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:55.111676 15995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:55.112879 15995 hybrid_clock.cc:648] HybridClock initialized: now 1781822395112824 us; error 49 us; skew 500 ppm
I20260618 22:39:55.113199 15995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:39:55.115562 15995 webserver.cc:460] Webserver started at http://127.5.213.194:42593/ using document root <none> and password file <none>
I20260618 22:39:55.116276 15995 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:55.116392 15995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:55.116705 15995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:55.119632 15995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "18ad4cbe9e604dd2b5eb44b4ae473668"
format_stamp: "Formatted at 2026-06-18 22:39:55 on dist-test-slave-lnj3"
server_key: "a5800cb44d17334515484a6018f39d78"
server_key_iv: "75ab06375f61b033db4d23156eb9d539"
server_key_version: "encryptionkey@0"
I20260618 22:39:55.120419 15995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "18ad4cbe9e604dd2b5eb44b4ae473668"
format_stamp: "Formatted at 2026-06-18 22:39:55 on dist-test-slave-lnj3"
server_key: "a5800cb44d17334515484a6018f39d78"
server_key_iv: "75ab06375f61b033db4d23156eb9d539"
server_key_version: "encryptionkey@0"
I20260618 22:39:55.125628 15995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260618 22:39:55.129066 16009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:55.130489 15995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260618 22:39:55.130679 15995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "18ad4cbe9e604dd2b5eb44b4ae473668"
format_stamp: "Formatted at 2026-06-18 22:39:55 on dist-test-slave-lnj3"
server_key: "a5800cb44d17334515484a6018f39d78"
server_key_iv: "75ab06375f61b033db4d23156eb9d539"
server_key_version: "encryptionkey@0"
I20260618 22:39:55.130896 15995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:55.136373 15995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:55.137112 15995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:55.137346 15995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:55.138195 15995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:55.139598 15995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:55.139693 15995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:55.139802 15995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:55.139870 15995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:55.169371 15995 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:46717
I20260618 22:39:55.169418 16121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:46717 every 8 connection(s)
I20260618 22:39:55.170660 15995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:39:55.176893 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 15995
I20260618 22:39:55.177057 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:39:55.177433 5975 external_mini_cluster.cc:1428] Setting key 8faa269e673d196f3f62604a32d9b752
I20260618 22:39:55.178529 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:55.173109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:49839 (local address 127.5.213.254:33519)
0618 22:39:55.173274 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:55.173279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:55.174695 (+ 1416us) server_negotiation.cc:407] Connection header received
0618 22:39:55.176355 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:55.176361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:55.176425 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:55.176510 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:55.177787 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:55.177793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:55.177798 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:55.177815 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:55.177846 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:55.177858 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:55.177969 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:55.178187 (+ 218us) server_negotiation.cc:299] Negotiation successful
0618 22:39:55.178238 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:39:55.179550 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:55.173640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33519 (local address 127.5.213.194:49839)
0618 22:39:55.174525 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:55.174563 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:55.176073 (+ 1510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:55.176762 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:55.176770 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:55.176861 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:55.177487 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:39:55.177490 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:55.177534 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:39:55.177630 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:55.177940 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:55.177952 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:55.177959 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 22:39:55.178249 (+ 290us) client_negotiation.cc:240] Negotiation successful
0618 22:39:55.178466 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":269,"threads_started":1}
I20260618 22:39:55.180186 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:33519
--builtin_ntp_servers=127.5.213.212:44525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260618 22:39:55.180913 16122 heartbeater.cc:347] Connected to a master server at 127.5.213.254:33519
I20260618 22:39:55.181202 16122 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:55.181734 16122 heartbeater.cc:511] Master 127.5.213.254:33519 requested a full tablet report, sending...
I20260618 22:39:55.182993 15806 ts_manager.cc:194] Registered new tserver with Master: 18ad4cbe9e604dd2b5eb44b4ae473668 (127.5.213.194:46717)
W20260618 22:39:55.490231 16125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:55.490530 16125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:55.490670 16125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:55.497884 16125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:55.497982 16125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:55.498095 16125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:39:55.507256 16125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:44525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:33519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:55.508436 16125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:55.510124 16125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:55.522614 16133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:55.522644 16130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:55.522644 16131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:55.523602 16125 server_base.cc:1034] running on GCE node
I20260618 22:39:55.524262 16125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:55.525218 16125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:55.526419 16125 hybrid_clock.cc:648] HybridClock initialized: now 1781822395526362 us; error 63 us; skew 500 ppm
I20260618 22:39:55.526753 16125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:39:55.528977 16125 webserver.cc:460] Webserver started at http://127.5.213.195:35943/ using document root <none> and password file <none>
I20260618 22:39:55.529639 16125 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:55.529735 16125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:55.530046 16125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:55.532963 16125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "2d184e2af7eb492d85cca18ecfe6eb8e"
format_stamp: "Formatted at 2026-06-18 22:39:55 on dist-test-slave-lnj3"
server_key: "31c7f4efef8e7ec78d986d59dea5e3de"
server_key_iv: "3e0c56b79f029d7d5e3a7a6dfc4d747b"
server_key_version: "encryptionkey@0"
I20260618 22:39:55.533732 16125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "2d184e2af7eb492d85cca18ecfe6eb8e"
format_stamp: "Formatted at 2026-06-18 22:39:55 on dist-test-slave-lnj3"
server_key: "31c7f4efef8e7ec78d986d59dea5e3de"
server_key_iv: "3e0c56b79f029d7d5e3a7a6dfc4d747b"
server_key_version: "encryptionkey@0"
I20260618 22:39:55.539054 16125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260618 22:39:55.542352 16139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:55.543709 16125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:39:55.543872 16125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "2d184e2af7eb492d85cca18ecfe6eb8e"
format_stamp: "Formatted at 2026-06-18 22:39:55 on dist-test-slave-lnj3"
server_key: "31c7f4efef8e7ec78d986d59dea5e3de"
server_key_iv: "3e0c56b79f029d7d5e3a7a6dfc4d747b"
server_key_version: "encryptionkey@0"
I20260618 22:39:55.544062 16125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:55.549505 16125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:55.550244 16125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:55.550485 16125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:55.551390 16125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:55.552692 16125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:55.552793 16125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:55.552892 16125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:55.552958 16125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:55.583117 16125 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:32979
I20260618 22:39:55.583191 16251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:32979 every 8 connection(s)
I20260618 22:39:55.584368 16125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:39:55.590134 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 16125
I20260618 22:39:55.590308 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:39:55.590839 5975 external_mini_cluster.cc:1428] Setting key 1beddec5c5a454eda7b24773f48fc9f4
I20260618 22:39:55.592564 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:55.586995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:37561 (local address 127.5.213.254:33519)
0618 22:39:55.587202 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:55.587208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:55.588701 (+ 1493us) server_negotiation.cc:407] Connection header received
0618 22:39:55.590301 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:55.590307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:55.590395 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:55.590503 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:55.591897 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:55.591904 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:55.591909 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:55.591927 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:55.591958 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:55.591970 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:55.592076 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:55.592306 (+ 230us) server_negotiation.cc:299] Negotiation successful
0618 22:39:55.592354 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:39:55.593643 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:55.587632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:33519 (local address 127.5.213.195:37561)
0618 22:39:55.588514 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:55.588587 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:55.590032 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:55.590722 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:55.590731 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:55.590843 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:55.591546 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:39:55.591550 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:55.591604 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:39:55.591702 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:55.592069 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:55.592080 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:55.592105 (+ 25us) client_negotiation.cc:769] Sending connection context
0618 22:39:55.592386 (+ 281us) client_negotiation.cc:240] Negotiation successful
0618 22:39:55.592635 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":260,"threads_started":1}
I20260618 22:39:55.594811 16252 heartbeater.cc:347] Connected to a master server at 127.5.213.254:33519
I20260618 22:39:55.595069 16252 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:55.595613 16252 heartbeater.cc:511] Master 127.5.213.254:33519 requested a full tablet report, sending...
I20260618 22:39:55.596822 15806 ts_manager.cc:194] Registered new tserver with Master: 2d184e2af7eb492d85cca18ecfe6eb8e (127.5.213.195:32979)
I20260618 22:39:55.605976 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260618 22:39:55.611783 15844 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:39:55.610327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60070 (local address 127.5.213.254:33519)
0618 22:39:55.610466 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:55.610472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:55.610957 (+ 485us) server_negotiation.cc:407] Connection header received
0618 22:39:55.611118 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:55.611123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:55.611172 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:55.611249 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:55.611580 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60070: BlockingRecv error: recv got EOF from 127.0.0.1:60070 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:39:55.612468 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 15865
I20260618 22:39:55.615425 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:55.613253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45264 (local address 127.5.213.193:38255)
0618 22:39:55.613673 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:55.613682 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:55.613712 (+ 30us) server_negotiation.cc:407] Connection header received
0618 22:39:55.613766 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:55.613772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:55.613965 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:55.614092 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:55.614912 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:55.614918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:55.614929 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:55.614942 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:55.614982 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:55.614999 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:55.615101 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:55.615144 (+ 43us) server_negotiation.cc:299] Negotiation successful
0618 22:39:55.615217 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":188,"threads_started":1}
I20260618 22:39:55.616534 15987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:55.695664 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 15865
I20260618 22:39:55.708448 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 15995
I20260618 22:39:55.711761 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:55.709324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38292 (local address 127.5.213.194:46717)
0618 22:39:55.709791 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:55.709798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:55.709825 (+ 27us) server_negotiation.cc:407] Connection header received
0618 22:39:55.709885 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:55.709891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:55.710103 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:55.710225 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:55.711194 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:55.711200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:55.711212 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:55.711224 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:55.711266 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:55.711298 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:55.711398 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:55.711465 (+ 67us) server_negotiation.cc:299] Negotiation successful
0618 22:39:55.711545 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":209,"threads_started":1}
I20260618 22:39:55.712956 16117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:55.792274 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 15995
I20260618 22:39:55.806481 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 16125
I20260618 22:39:55.810205 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:55.807467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46450 (local address 127.5.213.195:32979)
0618 22:39:55.808087 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:55.808100 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:55.808130 (+ 30us) server_negotiation.cc:407] Connection header received
0618 22:39:55.808207 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:55.808214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:55.808424 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:55.808549 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:55.809529 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:55.809535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:55.809547 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:39:55.809563 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:55.809603 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:39:55.809617 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:55.809749 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:55.809802 (+ 53us) server_negotiation.cc:299] Negotiation successful
0618 22:39:55.809886 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":337,"threads_started":1}
I20260618 22:39:55.811568 16247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:55.890715 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 16125
I20260618 22:39:55.902832 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 15775
I20260618 22:39:55.904022 15836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:56.007228 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 15775
2026-06-18T22:39:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2212 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:39:56 dist-test-slave-lnj3 krb5kdc[16270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:39:56 dist-test-slave-lnj3 krb5kdc[16270](info): set up 2 sockets
Jun 18 22:39:56 dist-test-slave-lnj3 krb5kdc[16270](info): commencing operation
krb5kdc: starting...
W20260618 22:39:57.031279 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:39:57 dist-test-slave-lnj3 krb5kdc[16270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:39:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:39:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:39:57.243446 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:36741
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:36741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:57.552205 16283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:57.552500 16283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:57.552609 16283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:57.559823 16283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:39:57.559942 16283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:57.560006 16283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:57.560066 16283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:39:57.560127 16283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:39:57.569876 16283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:36741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:36741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:57.571185 16283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:57.572441 16283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:57.583741 16289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:57.583818 16288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:57.583737 16291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:57.584453 16283 server_base.cc:1034] running on GCE node
I20260618 22:39:57.585178 16283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:57.586472 16283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:57.587667 16283 hybrid_clock.cc:648] HybridClock initialized: now 1781822397587591 us; error 70 us; skew 500 ppm
I20260618 22:39:57.588011 16283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:57 dist-test-slave-lnj3 krb5kdc[16270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822397, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:57.590894 16283 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:39:57.592180 16283 webserver.cc:460] Webserver started at http://127.5.213.254:46563/ using document root <none> and password file <none>
I20260618 22:39:57.592796 16283 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:57.592890 16283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:57.593117 16283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:57.595934 16283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "37cf4b28cbb7466e9bbab9c157dfb621"
format_stamp: "Formatted at 2026-06-18 22:39:57 on dist-test-slave-lnj3"
server_key: "e67377c1e484a3fffce5ceda1abfcc99"
server_key_iv: "cfd1cfc0f367b0433d810d62ea532c0f"
server_key_version: "encryptionkey@0"
I20260618 22:39:57.596616 16283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "37cf4b28cbb7466e9bbab9c157dfb621"
format_stamp: "Formatted at 2026-06-18 22:39:57 on dist-test-slave-lnj3"
server_key: "e67377c1e484a3fffce5ceda1abfcc99"
server_key_iv: "cfd1cfc0f367b0433d810d62ea532c0f"
server_key_version: "encryptionkey@0"
I20260618 22:39:57.601529 16283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260618 22:39:57.604722 16298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:57.606076 16283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260618 22:39:57.606201 16283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "37cf4b28cbb7466e9bbab9c157dfb621"
format_stamp: "Formatted at 2026-06-18 22:39:57 on dist-test-slave-lnj3"
server_key: "e67377c1e484a3fffce5ceda1abfcc99"
server_key_iv: "cfd1cfc0f367b0433d810d62ea532c0f"
server_key_version: "encryptionkey@0"
I20260618 22:39:57.606369 16283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:57.630724 16283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:57.634526 16283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:57.634851 16283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:57.651275 16283 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:36741
I20260618 22:39:57.651306 16349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:36741 every 8 connection(s)
I20260618 22:39:57.652596 16283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:39:57.656075 16350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:57.662032 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 16283
I20260618 22:39:57.662225 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:39:57.662834 5975 external_mini_cluster.cc:1428] Setting key cc595debceae89d5d6cfe4f03095e6b3
I20260618 22:39:57.666074 16350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621: Bootstrap starting.
I20260618 22:39:57.669649 16350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:57.670821 16350 log.cc:824] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:57.673664 16350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621: No bootstrap required, opened a new log
Jun 18 22:39:57 dist-test-slave-lnj3 krb5kdc[16270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:39:57.682487 16350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cf4b28cbb7466e9bbab9c157dfb621" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36741 } }
I20260618 22:39:57.682893 16350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:57.682998 16350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37cf4b28cbb7466e9bbab9c157dfb621, State: Initialized, Role: FOLLOWER
I20260618 22:39:57.683590 16350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cf4b28cbb7466e9bbab9c157dfb621" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36741 } }
I20260618 22:39:57.683810 16350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:39:57.683969 16350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:39:57.684136 16350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:57.685532 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:57.664722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58950 (local address 127.5.213.254:36741)
0618 22:39:57.665433 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:57.665444 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:57.665484 (+ 40us) server_negotiation.cc:407] Connection header received
0618 22:39:57.667304 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:57.667327 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:57.667732 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:57.668057 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:57.669823 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:57.671206 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:57.672743 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:57.673388 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:57.673510 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:57.678104 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:57.678112 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:57.678127 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:57.678169 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:57.681392 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:57.682151 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:57.682160 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:57.682169 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:57.682343 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:57.682705 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:57.682710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:57.682713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:57.683365 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:57.683651 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:57.684052 (+ 401us) server_negotiation.cc:299] Negotiation successful
0618 22:39:57.684462 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":319,"threads_started":1}
I20260618 22:39:57.686488 16350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cf4b28cbb7466e9bbab9c157dfb621" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36741 } }
I20260618 22:39:57.687064 16350 leader_election.cc:302] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37cf4b28cbb7466e9bbab9c157dfb621; no voters:
I20260618 22:39:57.687696 16350 leader_election.cc:288] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:39:57.687831 16355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:57.689095 16355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621 [term 1 LEADER]: Becoming Leader. State: Replica: 37cf4b28cbb7466e9bbab9c157dfb621, State: Running, Role: LEADER
I20260618 22:39:57.689666 16350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:39:57.689556 16355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cf4b28cbb7466e9bbab9c157dfb621" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36741 } }
I20260618 22:39:57.692071 16356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37cf4b28cbb7466e9bbab9c157dfb621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cf4b28cbb7466e9bbab9c157dfb621" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36741 } } }
I20260618 22:39:57.692044 16357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37cf4b28cbb7466e9bbab9c157dfb621. Latest consensus state: current_term: 1 leader_uuid: "37cf4b28cbb7466e9bbab9c157dfb621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cf4b28cbb7466e9bbab9c157dfb621" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36741 } } }
I20260618 22:39:57.692410 16357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:57.692410 16356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621 [sys.catalog]: This master's current role is: LEADER
I20260618 22:39:57.694006 16361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:39:57.698366 16361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:39:57.706269 16361 catalog_manager.cc:1347] Generated new cluster ID: d9f4a3fad3854f0b87f77fe0d8db584e
I20260618 22:39:57.706369 16361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:39:57.740082 16361 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:39:57.742010 16361 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:39:57.749692 16361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621: Generated new TSK 0
I20260618 22:39:57.750715 16361 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
I20260618 22:39:57.797737 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36741
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:58.119194 16376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:58.119475 16376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:58.119611 16376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:58.126772 16376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:58.126878 16376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:58.127018 16376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:39:58.136621 16376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:58.137861 16376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:58.139194 16376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:58.151453 16381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:58.151415 16382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:58.151479 16384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:58.152324 16376 server_base.cc:1034] running on GCE node
I20260618 22:39:58.152853 16376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:58.153692 16376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:58.154908 16376 hybrid_clock.cc:648] HybridClock initialized: now 1781822398154846 us; error 50 us; skew 500 ppm
I20260618 22:39:58.155229 16376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:58 dist-test-slave-lnj3 krb5kdc[16270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822398, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:58.158160 16376 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:39:58.159511 16376 webserver.cc:460] Webserver started at http://127.5.213.193:38159/ using document root <none> and password file <none>
I20260618 22:39:58.160141 16376 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:58.160238 16376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:58.160470 16376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:58.163277 16376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "710145f2a6454aba8553fac80080746e"
format_stamp: "Formatted at 2026-06-18 22:39:58 on dist-test-slave-lnj3"
server_key: "ba6e26dc66fdad5442a3d8a986369305"
server_key_iv: "2ed1b54f2b133fb2594313d72fe7c00b"
server_key_version: "encryptionkey@0"
I20260618 22:39:58.163972 16376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "710145f2a6454aba8553fac80080746e"
format_stamp: "Formatted at 2026-06-18 22:39:58 on dist-test-slave-lnj3"
server_key: "ba6e26dc66fdad5442a3d8a986369305"
server_key_iv: "2ed1b54f2b133fb2594313d72fe7c00b"
server_key_version: "encryptionkey@0"
I20260618 22:39:58.168836 16376 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:39:58.172297 16391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:58.173604 16376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260618 22:39:58.173741 16376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "710145f2a6454aba8553fac80080746e"
format_stamp: "Formatted at 2026-06-18 22:39:58 on dist-test-slave-lnj3"
server_key: "ba6e26dc66fdad5442a3d8a986369305"
server_key_iv: "2ed1b54f2b133fb2594313d72fe7c00b"
server_key_version: "encryptionkey@0"
I20260618 22:39:58.173920 16376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:58.206976 16376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:58.211056 16376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:58.211313 16376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:58.212250 16376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:58.213529 16376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:58.213613 16376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:58.213729 16376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:58.213778 16376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:58.242849 16376 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:34271
I20260618 22:39:58.242903 16503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:34271 every 8 connection(s)
I20260618 22:39:58.244107 16376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:39:58.253625 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 16376
I20260618 22:39:58.253854 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:39:58.254551 5975 external_mini_cluster.cc:1428] Setting key 90440cf64cd7877e6889f283ac1cb92f
Jun 18 22:39:58 dist-test-slave-lnj3 krb5kdc[16270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822398, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:39:58.263568 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:58.246370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:59589 (local address 127.5.213.254:36741)
0618 22:39:58.246825 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:58.246831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:58.247790 (+ 959us) server_negotiation.cc:407] Connection header received
0618 22:39:58.249493 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:58.249498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:58.249564 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:58.249656 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:58.251217 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:58.251929 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:58.253434 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:58.254018 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:58.254147 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:58.257511 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:58.257518 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:58.257523 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:58.257562 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:58.260511 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:58.261212 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:58.261217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:58.261220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:58.261351 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:58.261757 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:58.261761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:58.261764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:58.262094 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:58.262224 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:58.263011 (+ 787us) server_negotiation.cc:299] Negotiation successful
0618 22:39:58.263231 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":196,"threads_started":1}
I20260618 22:39:58.264405 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:58.246984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:36741 (local address 127.5.213.193:59589)
0618 22:39:58.247636 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:58.247676 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:58.249256 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:58.249932 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:58.249943 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:58.250467 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:58.251069 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:58.251087 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:58.252063 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:58.252067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:58.253295 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:58.253306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:58.254230 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:58.254235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:58.254431 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:58.255079 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:58.255114 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:58.257302 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:58.260665 (+ 3363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:58.260671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:58.260684 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:58.261100 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:58.261448 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:58.261451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:58.261453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:58.261659 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:58.262242 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:58.262248 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:58.262569 (+ 321us) client_negotiation.cc:769] Sending connection context
0618 22:39:58.262891 (+ 322us) client_negotiation.cc:240] Negotiation successful
0618 22:39:58.263276 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":247,"threads_started":1}
I20260618 22:39:58.265847 16504 heartbeater.cc:347] Connected to a master server at 127.5.213.254:36741
I20260618 22:39:58.266119 16504 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:58.266852 16504 heartbeater.cc:511] Master 127.5.213.254:36741 requested a full tablet report, sending...
I20260618 22:39:58.268764 16315 ts_manager.cc:194] Registered new tserver with Master: 710145f2a6454aba8553fac80080746e (127.5.213.193:34271)
I20260618 22:39:58.270116 16315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:59589
WARNING: no policy specified for kudu/127.5.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
I20260618 22:39:58.303870 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36741
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:58.615496 16510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:58.615833 16510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:58.615962 16510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:58.623169 16510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:58.623294 16510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:58.623435 16510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:39:58.633571 16510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:58.634922 16510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:58.636201 16510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:58.648792 16515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:58.648875 16516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:58.648900 16518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:58.649811 16510 server_base.cc:1034] running on GCE node
I20260618 22:39:58.650401 16510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:58.651469 16510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:58.652671 16510 hybrid_clock.cc:648] HybridClock initialized: now 1781822398652615 us; error 48 us; skew 500 ppm
I20260618 22:39:58.652972 16510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:58 dist-test-slave-lnj3 krb5kdc[16270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822398, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:58.655937 16510 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:39:58.657364 16510 webserver.cc:460] Webserver started at http://127.5.213.194:42917/ using document root <none> and password file <none>
I20260618 22:39:58.658015 16510 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:58.658115 16510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:58.658381 16510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:58.661235 16510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "d58c5d7f64364dcf8f227dab3f344ed1"
format_stamp: "Formatted at 2026-06-18 22:39:58 on dist-test-slave-lnj3"
server_key: "3a6bb72cd9a82693666e8c5aa64603bc"
server_key_iv: "fa7e0fc4fd99f14bfbfd4f2232dc4c5f"
server_key_version: "encryptionkey@0"
I20260618 22:39:58.661975 16510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "d58c5d7f64364dcf8f227dab3f344ed1"
format_stamp: "Formatted at 2026-06-18 22:39:58 on dist-test-slave-lnj3"
server_key: "3a6bb72cd9a82693666e8c5aa64603bc"
server_key_iv: "fa7e0fc4fd99f14bfbfd4f2232dc4c5f"
server_key_version: "encryptionkey@0"
I20260618 22:39:58.666894 16510 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260618 22:39:58.669993 16525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:58.671345 16510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:39:58.671474 16510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "d58c5d7f64364dcf8f227dab3f344ed1"
format_stamp: "Formatted at 2026-06-18 22:39:58 on dist-test-slave-lnj3"
server_key: "3a6bb72cd9a82693666e8c5aa64603bc"
server_key_iv: "fa7e0fc4fd99f14bfbfd4f2232dc4c5f"
server_key_version: "encryptionkey@0"
I20260618 22:39:58.671638 16510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:58.689888 16510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:58.693825 16510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:58.694089 16510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:58.695001 16510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:58.696295 16510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:58.696370 16510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:58.696487 16510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:58.696537 16510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:58.725301 16510 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:36855
I20260618 22:39:58.725333 16637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:36855 every 8 connection(s)
I20260618 22:39:58.726485 16510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:39:58.733471 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 16510
I20260618 22:39:58.733659 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:39:58.734146 5975 external_mini_cluster.cc:1428] Setting key 10419d06f3820cb94c44a6708c6c2996
Jun 18 22:39:58 dist-test-slave-lnj3 krb5kdc[16270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822398, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:39:58.746800 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:58.728834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:35245 (local address 127.5.213.254:36741)
0618 22:39:58.729027 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:58.729033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:58.730249 (+ 1216us) server_negotiation.cc:407] Connection header received
0618 22:39:58.732416 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:58.732421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:58.732512 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:58.732596 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:58.734541 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:58.735447 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:58.737523 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:58.737940 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:58.738032 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:58.741532 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:58.741536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:58.741539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:58.741568 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:58.743999 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:58.744648 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:58.744654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:58.744658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:58.744803 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:58.745153 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:58.745157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:58.745160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:58.745469 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:58.745639 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:58.746230 (+ 591us) server_negotiation.cc:299] Negotiation successful
0618 22:39:58.746436 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260618 22:39:58.747661 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:58.729345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:36741 (local address 127.5.213.194:35245)
0618 22:39:58.730061 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:58.730108 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:58.732167 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:58.732874 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:58.732882 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:58.733586 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:58.734367 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:58.734395 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:58.735603 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:58.735608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:58.737381 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:58.737397 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:58.738147 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:58.738153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:58.738415 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:58.739085 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:58.739110 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:58.741336 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:58.744121 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:58.744129 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:58.744142 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:58.744531 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:58.744900 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:58.744903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:58.744905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:58.745067 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:58.745583 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:58.745591 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:58.745908 (+ 317us) client_negotiation.cc:769] Sending connection context
0618 22:39:58.746125 (+ 217us) client_negotiation.cc:240] Negotiation successful
0618 22:39:58.746484 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":245,"threads_started":1}
I20260618 22:39:58.748987 16638 heartbeater.cc:347] Connected to a master server at 127.5.213.254:36741
I20260618 22:39:58.749271 16638 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:58.750011 16638 heartbeater.cc:511] Master 127.5.213.254:36741 requested a full tablet report, sending...
I20260618 22:39:58.751384 16315 ts_manager.cc:194] Registered new tserver with Master: d58c5d7f64364dcf8f227dab3f344ed1 (127.5.213.194:36855)
I20260618 22:39:58.752120 16315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:35245
WARNING: no policy specified for kudu/127.5.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
I20260618 22:39:58.779812 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36741
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:39:59.089326 16643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:39:59.089640 16643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:39:59.089746 16643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:39:59.096948 16643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:39:59.097059 16643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:39:59.097167 16643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:39:59.106657 16643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:39:59.107857 16643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:39:59.109089 16643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:39:59.121344 16649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:59.121336 16651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:39:59.121510 16648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:39:59.122161 16643 server_base.cc:1034] running on GCE node
I20260618 22:39:59.122790 16643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:39:59.123708 16643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:39:59.124907 16643 hybrid_clock.cc:648] HybridClock initialized: now 1781822399124852 us; error 49 us; skew 500 ppm
I20260618 22:39:59.125221 16643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:39:59 dist-test-slave-lnj3 krb5kdc[16270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822399, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:39:59.128255 16643 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:39:59.129601 16643 webserver.cc:460] Webserver started at http://127.5.213.195:33597/ using document root <none> and password file <none>
I20260618 22:39:59.130271 16643 fs_manager.cc:359] Metadata directory not provided
I20260618 22:39:59.130384 16643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:39:59.130686 16643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:39:59.133478 16643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "abbc2992843a4456ad6d9c6d8ce6bfbd"
format_stamp: "Formatted at 2026-06-18 22:39:59 on dist-test-slave-lnj3"
server_key: "fc62977cde42b6d7a1f913d379a95cf1"
server_key_iv: "3464375ca95155ff964dc0bb91183f5b"
server_key_version: "encryptionkey@0"
I20260618 22:39:59.134223 16643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "abbc2992843a4456ad6d9c6d8ce6bfbd"
format_stamp: "Formatted at 2026-06-18 22:39:59 on dist-test-slave-lnj3"
server_key: "fc62977cde42b6d7a1f913d379a95cf1"
server_key_iv: "3464375ca95155ff964dc0bb91183f5b"
server_key_version: "encryptionkey@0"
I20260618 22:39:59.139169 16643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260618 22:39:59.142450 16658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:59.143800 16643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:39:59.143951 16643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "abbc2992843a4456ad6d9c6d8ce6bfbd"
format_stamp: "Formatted at 2026-06-18 22:39:59 on dist-test-slave-lnj3"
server_key: "fc62977cde42b6d7a1f913d379a95cf1"
server_key_iv: "3464375ca95155ff964dc0bb91183f5b"
server_key_version: "encryptionkey@0"
I20260618 22:39:59.144133 16643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:39:59.187372 16643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:39:59.191591 16643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:39:59.191831 16643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:39:59.192699 16643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:39:59.193948 16643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:39:59.194034 16643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:59.194141 16643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:39:59.194193 16643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:39:59.222858 16643 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:34033
I20260618 22:39:59.222893 16770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:34033 every 8 connection(s)
I20260618 22:39:59.224042 16643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:39:59.230368 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 16643
I20260618 22:39:59.230541 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:39:59.231016 5975 external_mini_cluster.cc:1428] Setting key d648bd56f4689cfd8bd339f9538376db
Jun 18 22:39:59 dist-test-slave-lnj3 krb5kdc[16270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822399, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:39:59.242502 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.226421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:37671 (local address 127.5.213.254:36741)
0618 22:39:59.226654 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:59.226661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:59.227758 (+ 1097us) server_negotiation.cc:407] Connection header received
0618 22:39:59.229476 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:59.229481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:59.229578 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:59.229661 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:59.231167 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.232098 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.234021 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.234413 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.234506 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:59.237614 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:59.237618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:59.237622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:59.237652 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:59.240050 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:59.240635 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:59.240639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:59.240641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:59.240741 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:59.241076 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:59.241079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:59.241081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:59.241322 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:59.241419 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:59.241988 (+ 569us) server_negotiation.cc:299] Negotiation successful
0618 22:39:59.242182 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260618 22:39:59.243383 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.226958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:36741 (local address 127.5.213.195:37671)
0618 22:39:59.227606 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:59.227650 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:59.229239 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:59.229879 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:59.229887 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:59.230363 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:39:59.231005 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:59.231022 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.232271 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.232276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:59.233892 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:59.233907 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.234574 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.234579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:59.234832 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:59.235394 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:39:59.235414 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:39:59.237407 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:39:59.240159 (+ 2752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:59.240166 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:59.240182 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:59.240548 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:59.240817 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:39:59.240820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:39:59.240822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:39:59.240981 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:39:59.241407 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:39:59.241412 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:39:59.241695 (+ 283us) client_negotiation.cc:769] Sending connection context
0618 22:39:59.241909 (+ 214us) client_negotiation.cc:240] Negotiation successful
0618 22:39:59.242273 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":247,"threads_started":1}
I20260618 22:39:59.244990 16771 heartbeater.cc:347] Connected to a master server at 127.5.213.254:36741
I20260618 22:39:59.245296 16771 heartbeater.cc:464] Registering TS with master...
I20260618 22:39:59.246071 16771 heartbeater.cc:511] Master 127.5.213.254:36741 requested a full tablet report, sending...
I20260618 22:39:59.247412 16315 ts_manager.cc:194] Registered new tserver with Master: abbc2992843a4456ad6d9c6d8ce6bfbd (127.5.213.195:34033)
I20260618 22:39:59.248322 16315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:37671
I20260618 22:39:59.257974 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:39:59.272975 16504 heartbeater.cc:503] Master 127.5.213.254:36741 was elected leader, sending a full tablet report...
I20260618 22:39:59.273097 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.262121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.5.213.254:36741)
0618 22:39:59.262301 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:59.262308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:59.262684 (+ 376us) server_negotiation.cc:407] Connection header received
0618 22:39:59.262823 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:59.262827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:59.262885 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:59.262962 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:59.264304 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.264998 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.266201 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.266590 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.266755 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:59.268690 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:59.268694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:59.268698 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:59.268733 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:59.271097 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:59.271582 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:59.271585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:59.271587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:59.271703 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:59.272004 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:59.272007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:59.272009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:59.272295 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:59.272460 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:59.272697 (+ 237us) server_negotiation.cc:299] Negotiation successful
0618 22:39:59.272863 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:39:59.277451 16314 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:39:59.284036 16314 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:39:59.306732 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.295674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:36855 (local address 127.0.0.1:34466)
0618 22:39:59.296915 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:59.296963 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:59.297149 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:59.298196 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:59.298205 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:59.298258 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:59.298683 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:59.298696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.300777 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.300784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:59.303391 (+ 2607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:59.303407 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.305651 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.305658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:59.305886 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:59.305938 (+ 52us) client_negotiation.cc:769] Sending connection context
0618 22:39:59.306074 (+ 136us) client_negotiation.cc:240] Negotiation successful
0618 22:39:59.306312 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"thread_start_us":832,"threads_started":1}
I20260618 22:39:59.307696 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.295838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34466 (local address 127.5.213.194:36855)
0618 22:39:59.296499 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:59.296510 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:59.297347 (+ 837us) server_negotiation.cc:407] Connection header received
0618 22:39:59.297464 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:59.297472 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:59.297739 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:59.297927 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:59.298875 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.300594 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.303572 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.305133 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.305284 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:59.305504 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:59.307150 (+ 1646us) server_negotiation.cc:299] Negotiation successful
0618 22:39:59.307355 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":359,"threads_started":1}
I20260618 22:39:59.312495 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.296199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:34033 (local address 127.0.0.1:40856)
0618 22:39:59.303442 (+ 7243us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:59.303458 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:59.303594 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:59.304669 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:59.304675 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:59.304704 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:59.305149 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:59.305160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.307247 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.307253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:59.309647 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:59.309663 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.311787 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.311794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:59.311933 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:59.311957 (+ 24us) client_negotiation.cc:769] Sending connection context
0618 22:39:59.312034 (+ 77us) client_negotiation.cc:240] Negotiation successful
0618 22:39:59.312163 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7140,"thread_start_us":7013,"threads_started":1}
I20260618 22:39:59.312573 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.296787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.5.213.195:34033)
0618 22:39:59.303171 (+ 6384us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:59.303179 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:59.303721 (+ 542us) server_negotiation.cc:407] Connection header received
0618 22:39:59.303838 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:59.303846 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:59.304129 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:59.304368 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:59.305368 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.307044 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.309839 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.311456 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.311570 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:59.311781 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:59.312101 (+ 320us) server_negotiation.cc:299] Negotiation successful
0618 22:39:59.312244 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6212,"thread_start_us":6069,"threads_started":1}
I20260618 22:39:59.312834 16573 tablet_service.cc:1467] Processing CreateTablet for tablet b99c19e9793e4c8abc0716dc17daba65 (DEFAULT_TABLE table=test-table [id=39a23480abc14372a8158c99f60d5149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:59.314239 16573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99c19e9793e4c8abc0716dc17daba65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:59.315680 16706 tablet_service.cc:1467] Processing CreateTablet for tablet b99c19e9793e4c8abc0716dc17daba65 (DEFAULT_TABLE table=test-table [id=39a23480abc14372a8158c99f60d5149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:59.316974 16706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99c19e9793e4c8abc0716dc17daba65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:59.321640 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.303963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:34271 (local address 127.0.0.1:48498)
0618 22:39:59.308445 (+ 4482us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:59.308464 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:59.308558 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:59.312968 (+ 4410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:59.312972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:59.312999 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:59.313144 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:59.313154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.314929 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.314934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:59.317771 (+ 2837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:59.317785 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.320938 (+ 3153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.320944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:59.321113 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:59.321146 (+ 33us) client_negotiation.cc:769] Sending connection context
0618 22:39:59.321230 (+ 84us) client_negotiation.cc:240] Negotiation successful
0618 22:39:59.321370 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4373,"thread_start_us":7374,"threads_started":1}
I20260618 22:39:59.322356 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.304067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.5.213.193:34271)
0618 22:39:59.311309 (+ 7242us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:59.311317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:59.311345 (+ 28us) server_negotiation.cc:407] Connection header received
0618 22:39:59.311430 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:59.311436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:59.312059 (+ 623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:59.312239 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:59.313313 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.314761 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.317952 (+ 3191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.320486 (+ 2534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.320611 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:59.320845 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:59.321919 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0618 22:39:59.322074 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7089,"thread_start_us":6955,"threads_started":1}
I20260618 22:39:59.326001 16439 tablet_service.cc:1467] Processing CreateTablet for tablet b99c19e9793e4c8abc0716dc17daba65 (DEFAULT_TABLE table=test-table [id=39a23480abc14372a8158c99f60d5149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:39:59.327445 16789 tablet_bootstrap.cc:492] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1: Bootstrap starting.
I20260618 22:39:59.327466 16439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99c19e9793e4c8abc0716dc17daba65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:39:59.329275 16790 tablet_bootstrap.cc:492] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd: Bootstrap starting.
I20260618 22:39:59.331229 16789 tablet_bootstrap.cc:654] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:59.332441 16789 log.cc:824] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:59.332551 16790 tablet_bootstrap.cc:654] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:59.333608 16790 log.cc:824] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:59.335361 16789 tablet_bootstrap.cc:492] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1: No bootstrap required, opened a new log
I20260618 22:39:59.335625 16789 ts_tablet_manager.cc:1397] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260618 22:39:59.336611 16790 tablet_bootstrap.cc:492] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd: No bootstrap required, opened a new log
I20260618 22:39:59.336849 16790 ts_tablet_manager.cc:1397] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260618 22:39:59.339751 16793 tablet_bootstrap.cc:492] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e: Bootstrap starting.
I20260618 22:39:59.342487 16793 tablet_bootstrap.cc:654] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e: Neither blocks nor log segments found. Creating new log.
I20260618 22:39:59.343497 16793 log.cc:824] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e: Log is configured to *not* fsync() on all Append() calls
I20260618 22:39:59.346488 16793 tablet_bootstrap.cc:492] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e: No bootstrap required, opened a new log
I20260618 22:39:59.346163 16789 raft_consensus.cc:348] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58c5d7f64364dcf8f227dab3f344ed1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 36855 } } peers { permanent_uuid: "abbc2992843a4456ad6d9c6d8ce6bfbd" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34033 } } peers { permanent_uuid: "710145f2a6454aba8553fac80080746e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34271 } }
I20260618 22:39:59.346657 16789 raft_consensus.cc:374] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:59.346832 16789 raft_consensus.cc:729] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d58c5d7f64364dcf8f227dab3f344ed1, State: Initialized, Role: FOLLOWER
I20260618 22:39:59.346941 16793 ts_tablet_manager.cc:1397] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260618 22:39:59.346906 16790 raft_consensus.cc:348] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58c5d7f64364dcf8f227dab3f344ed1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 36855 } } peers { permanent_uuid: "abbc2992843a4456ad6d9c6d8ce6bfbd" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34033 } } peers { permanent_uuid: "710145f2a6454aba8553fac80080746e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34271 } }
I20260618 22:39:59.347486 16790 raft_consensus.cc:374] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:59.347615 16790 raft_consensus.cc:729] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abbc2992843a4456ad6d9c6d8ce6bfbd, State: Initialized, Role: FOLLOWER
I20260618 22:39:59.347575 16789 consensus_queue.cc:260] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58c5d7f64364dcf8f227dab3f344ed1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 36855 } } peers { permanent_uuid: "abbc2992843a4456ad6d9c6d8ce6bfbd" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34033 } } peers { permanent_uuid: "710145f2a6454aba8553fac80080746e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34271 } }
I20260618 22:39:59.348358 16790 consensus_queue.cc:260] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58c5d7f64364dcf8f227dab3f344ed1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 36855 } } peers { permanent_uuid: "abbc2992843a4456ad6d9c6d8ce6bfbd" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34033 } } peers { permanent_uuid: "710145f2a6454aba8553fac80080746e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34271 } }
I20260618 22:39:59.349535 16771 heartbeater.cc:503] Master 127.5.213.254:36741 was elected leader, sending a full tablet report...
I20260618 22:39:59.350371 16790 ts_tablet_manager.cc:1428] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260618 22:39:59.351799 16638 heartbeater.cc:503] Master 127.5.213.254:36741 was elected leader, sending a full tablet report...
I20260618 22:39:59.352437 16789 ts_tablet_manager.cc:1428] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260618 22:39:59.355772 16793 raft_consensus.cc:348] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58c5d7f64364dcf8f227dab3f344ed1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 36855 } } peers { permanent_uuid: "abbc2992843a4456ad6d9c6d8ce6bfbd" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34033 } } peers { permanent_uuid: "710145f2a6454aba8553fac80080746e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34271 } }
I20260618 22:39:59.356175 16793 raft_consensus.cc:374] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:39:59.356266 16793 raft_consensus.cc:729] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 710145f2a6454aba8553fac80080746e, State: Initialized, Role: FOLLOWER
I20260618 22:39:59.356854 16793 consensus_queue.cc:260] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58c5d7f64364dcf8f227dab3f344ed1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 36855 } } peers { permanent_uuid: "abbc2992843a4456ad6d9c6d8ce6bfbd" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34033 } } peers { permanent_uuid: "710145f2a6454aba8553fac80080746e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34271 } }
I20260618 22:39:59.358481 16793 ts_tablet_manager.cc:1428] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e: Time spent starting tablet: real 0.011s user 0.005s sys 0.006s
W20260618 22:39:59.475592 16772 tablet.cc:2369] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:39:59.475931 16796 raft_consensus.cc:482] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:39:59.476199 16796 raft_consensus.cc:504] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58c5d7f64364dcf8f227dab3f344ed1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 36855 } } peers { permanent_uuid: "abbc2992843a4456ad6d9c6d8ce6bfbd" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34033 } } peers { permanent_uuid: "710145f2a6454aba8553fac80080746e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34271 } }
I20260618 22:39:59.477885 16796 leader_election.cc:288] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d58c5d7f64364dcf8f227dab3f344ed1 (127.5.213.194:36855), 710145f2a6454aba8553fac80080746e (127.5.213.193:34271)
W20260618 22:39:59.478420 16639 tablet.cc:2369] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:39:59.483969 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.478337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:59291 (local address 127.5.213.193:34271)
0618 22:39:59.478509 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:59.478514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:59.478533 (+ 19us) server_negotiation.cc:407] Connection header received
0618 22:39:59.478803 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:59.478809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:59.478893 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:59.478988 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:59.479511 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.480470 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.482165 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.483197 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.483312 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:59.483363 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:59.483674 (+ 311us) server_negotiation.cc:299] Negotiation successful
0618 22:39:59.483780 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 22:39:59.484077 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.478119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:34271 (local address 127.5.213.195:59291)
0618 22:39:59.478447 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:59.478467 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:59.478593 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:59.479145 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:59.479150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:59.479177 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:59.479353 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:59.479364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.480594 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.480599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:59.482015 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:59.482027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.483335 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.483341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:59.483524 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:59.483558 (+ 34us) client_negotiation.cc:769] Sending connection context
0618 22:39:59.483656 (+ 98us) client_negotiation.cc:240] Negotiation successful
0618 22:39:59.483806 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204}
I20260618 22:39:59.484750 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.478119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:36855 (local address 127.5.213.195:38935)
0618 22:39:59.479283 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:59.479303 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:39:59.479399 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:39:59.479660 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:39:59.479663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:39:59.479685 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:39:59.480017 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:59.480028 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.481159 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.481162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:59.482768 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:39:59.482782 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.484063 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.484073 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:39:59.484241 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:59.484272 (+ 31us) client_negotiation.cc:769] Sending connection context
0618 22:39:59.484395 (+ 123us) client_negotiation.cc:240] Negotiation successful
0618 22:39:59.484486 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"mutex_wait_us":531,"thread_start_us":286,"threads_started":1}
I20260618 22:39:59.484807 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.478297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:38935 (local address 127.5.213.194:36855)
0618 22:39:59.478496 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:59.478506 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:59.479387 (+ 881us) server_negotiation.cc:407] Connection header received
0618 22:39:59.479510 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:59.479513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:59.479566 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:59.479631 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:39:59.480153 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.481009 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.482906 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.483864 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.484010 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:59.484078 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:59.484443 (+ 365us) server_negotiation.cc:299] Negotiation successful
0618 22:39:59.484575 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260618 22:39:59.485230 16459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99c19e9793e4c8abc0716dc17daba65" candidate_uuid: "abbc2992843a4456ad6d9c6d8ce6bfbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "710145f2a6454aba8553fac80080746e" is_pre_election: true
I20260618 22:39:59.485725 16459 raft_consensus.cc:2454] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abbc2992843a4456ad6d9c6d8ce6bfbd in term 0.
I20260618 22:39:59.485702 16593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99c19e9793e4c8abc0716dc17daba65" candidate_uuid: "abbc2992843a4456ad6d9c6d8ce6bfbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58c5d7f64364dcf8f227dab3f344ed1" is_pre_election: true
I20260618 22:39:59.486179 16593 raft_consensus.cc:2454] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abbc2992843a4456ad6d9c6d8ce6bfbd in term 0.
I20260618 22:39:59.486321 16660 leader_election.cc:302] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710145f2a6454aba8553fac80080746e, abbc2992843a4456ad6d9c6d8ce6bfbd; no voters:
I20260618 22:39:59.486686 16796 raft_consensus.cc:2781] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:39:59.486825 16796 raft_consensus.cc:482] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:39:59.486927 16796 raft_consensus.cc:3037] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:59.488981 16796 raft_consensus.cc:504] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58c5d7f64364dcf8f227dab3f344ed1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 36855 } } peers { permanent_uuid: "abbc2992843a4456ad6d9c6d8ce6bfbd" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34033 } } peers { permanent_uuid: "710145f2a6454aba8553fac80080746e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34271 } }
I20260618 22:39:59.489506 16796 leader_election.cc:288] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [CANDIDATE]: Term 1 election: Requested vote from peers d58c5d7f64364dcf8f227dab3f344ed1 (127.5.213.194:36855), 710145f2a6454aba8553fac80080746e (127.5.213.193:34271)
I20260618 22:39:59.489861 16593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99c19e9793e4c8abc0716dc17daba65" candidate_uuid: "abbc2992843a4456ad6d9c6d8ce6bfbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58c5d7f64364dcf8f227dab3f344ed1"
I20260618 22:39:59.489969 16459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99c19e9793e4c8abc0716dc17daba65" candidate_uuid: "abbc2992843a4456ad6d9c6d8ce6bfbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "710145f2a6454aba8553fac80080746e"
I20260618 22:39:59.490026 16593 raft_consensus.cc:3037] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:59.490132 16459 raft_consensus.cc:3037] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:39:59.492158 16593 raft_consensus.cc:2454] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abbc2992843a4456ad6d9c6d8ce6bfbd in term 1.
I20260618 22:39:59.492164 16459 raft_consensus.cc:2454] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abbc2992843a4456ad6d9c6d8ce6bfbd in term 1.
I20260618 22:39:59.492532 16661 leader_election.cc:302] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abbc2992843a4456ad6d9c6d8ce6bfbd, d58c5d7f64364dcf8f227dab3f344ed1; no voters:
I20260618 22:39:59.492838 16796 raft_consensus.cc:2781] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:39:59.493228 16796 raft_consensus.cc:686] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [term 1 LEADER]: Becoming Leader. State: Replica: abbc2992843a4456ad6d9c6d8ce6bfbd, State: Running, Role: LEADER
I20260618 22:39:59.493639 16796 consensus_queue.cc:237] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58c5d7f64364dcf8f227dab3f344ed1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 36855 } } peers { permanent_uuid: "abbc2992843a4456ad6d9c6d8ce6bfbd" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34033 } } peers { permanent_uuid: "710145f2a6454aba8553fac80080746e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34271 } }
W20260618 22:39:59.496079 16505 tablet.cc:2369] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:39:59.498175 16315 catalog_manager.cc:5495] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd reported cstate change: term changed from 0 to 1, leader changed from <none> to abbc2992843a4456ad6d9c6d8ce6bfbd (127.5.213.195). New cstate: current_term: 1 leader_uuid: "abbc2992843a4456ad6d9c6d8ce6bfbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58c5d7f64364dcf8f227dab3f344ed1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 36855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abbc2992843a4456ad6d9c6d8ce6bfbd" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710145f2a6454aba8553fac80080746e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34271 } health_report { overall_health: UNKNOWN } } }
I20260618 22:39:59.555500 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.550110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.5.213.195:34033)
0618 22:39:59.550310 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:59.550316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:59.550402 (+ 86us) server_negotiation.cc:407] Connection header received
0618 22:39:59.550592 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:59.550596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:59.550693 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:59.550815 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:39:59.551275 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.552047 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.553394 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.553825 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.553928 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:59.554349 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:39:59.554968 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:39:59.555072 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:59.555169 (+ 97us) server_negotiation.cc:299] Negotiation successful
0618 22:39:59.555245 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:39:59.564000 16593 raft_consensus.cc:1261] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1 [term 1 FOLLOWER]: Refusing update from remote peer abbc2992843a4456ad6d9c6d8ce6bfbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:39:59.564009 16459 raft_consensus.cc:1261] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e [term 1 FOLLOWER]: Refusing update from remote peer abbc2992843a4456ad6d9c6d8ce6bfbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:39:59.565073 16796 consensus_queue.cc:1035] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d58c5d7f64364dcf8f227dab3f344ed1" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 36855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:59.565447 16799 consensus_queue.cc:1035] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "710145f2a6454aba8553fac80080746e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:39:59.575382 16802 mvcc.cc:205] Tried to move back new op lower bound from 7298344548603686912 to 7298344548329181184. Current Snapshot: MvccSnapshot[applied={T|T < 7298344548603686912}]
I20260618 22:39:59.578263 16803 mvcc.cc:205] Tried to move back new op lower bound from 7298344548603686912 to 7298344548329181184. Current Snapshot: MvccSnapshot[applied={T|T < 7298344548603686912}]
I20260618 22:39:59.590587 16315 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:39:59.590875 16315 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:39:59.594489 16315 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37cf4b28cbb7466e9bbab9c157dfb621: Sending DeleteTablet for 3 replicas of tablet b99c19e9793e4c8abc0716dc17daba65
I20260618 22:39:59.595750 16706 tablet_service.cc:1514] Processing DeleteTablet for tablet b99c19e9793e4c8abc0716dc17daba65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:59 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:40856
I20260618 22:39:59.595937 16573 tablet_service.cc:1514] Processing DeleteTablet for tablet b99c19e9793e4c8abc0716dc17daba65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:59 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:34466
I20260618 22:39:59.596618 16439 tablet_service.cc:1514] Processing DeleteTablet for tablet b99c19e9793e4c8abc0716dc17daba65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:39:59 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:48498
I20260618 22:39:59.596706 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 16376
I20260618 22:39:59.605458 16814 tablet_replica.cc:330] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e: stopping tablet replica
I20260618 22:39:59.606673 16814 raft_consensus.cc:2229] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:39:59.607925 16814 raft_consensus.cc:2258] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:59.608842 16812 tablet_replica.cc:330] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd: stopping tablet replica
I20260618 22:39:59.609756 16812 raft_consensus.cc:2229] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:39:59.611866 16814 ts_tablet_manager.cc:1905] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:59.612219 16812 raft_consensus.cc:2258] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:59.613332 16813 tablet_replica.cc:330] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1: stopping tablet replica
I20260618 22:39:59.613840 16813 raft_consensus.cc:2229] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:39:59.614362 16813 raft_consensus.cc:2258] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:39:59.616034 16812 ts_tablet_manager.cc:1905] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:39:59.616443 16813 ts_tablet_manager.cc:1905] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 18 22:39:59 dist-test-slave-lnj3 krb5kdc[16270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:39:59.623027 16813 ts_tablet_manager.cc:1918] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:39:59.624154 16812 ts_tablet_manager.cc:1918] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:39:59.624397 16812 log.cc:1196] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/b99c19e9793e4c8abc0716dc17daba65
I20260618 22:39:59.624977 16812 ts_tablet_manager.cc:1939] T b99c19e9793e4c8abc0716dc17daba65 P abbc2992843a4456ad6d9c6d8ce6bfbd: Deleting consensus metadata
I20260618 22:39:59.626803 16300 catalog_manager.cc:4841] TS abbc2992843a4456ad6d9c6d8ce6bfbd (127.5.213.195:34033): tablet b99c19e9793e4c8abc0716dc17daba65 (table test-table [id=39a23480abc14372a8158c99f60d5149]) successfully deleted
I20260618 22:39:59.627691 16813 log.cc:1196] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/b99c19e9793e4c8abc0716dc17daba65
I20260618 22:39:59.628196 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.598107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48506 (local address 127.5.213.193:34271)
0618 22:39:59.598279 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:59.598285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:59.608391 (+ 10106us) server_negotiation.cc:407] Connection header received
0618 22:39:59.608874 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:59.608880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:59.608963 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:59.609038 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:59.611542 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.612566 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.614718 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.615472 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.615567 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:59.620633 (+ 5066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:59.620646 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:59.620664 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:59.620708 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:59.624386 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:59.625199 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:59.625209 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:59.625212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:59.625371 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:59.625825 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:59.625830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:59.625833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:59.626301 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:59.627161 (+ 860us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:59.627330 (+ 169us) server_negotiation.cc:299] Negotiation successful
0618 22:39:59.627723 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:39:59.628253 16813 ts_tablet_manager.cc:1939] T b99c19e9793e4c8abc0716dc17daba65 P d58c5d7f64364dcf8f227dab3f344ed1: Deleting consensus metadata
I20260618 22:39:59.628217 16814 ts_tablet_manager.cc:1918] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:39:59.629408 16814 log.cc:1196] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/b99c19e9793e4c8abc0716dc17daba65
I20260618 22:39:59.629894 16499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:59.629940 16814 ts_tablet_manager.cc:1939] T b99c19e9793e4c8abc0716dc17daba65 P 710145f2a6454aba8553fac80080746e: Deleting consensus metadata
I20260618 22:39:59.629963 16301 catalog_manager.cc:4841] TS d58c5d7f64364dcf8f227dab3f344ed1 (127.5.213.194:36855): tablet b99c19e9793e4c8abc0716dc17daba65 (table test-table [id=39a23480abc14372a8158c99f60d5149]) successfully deleted
I20260618 22:39:59.743512 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 16376
W20260618 22:39:59.758118 16300 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.213.193:34271 (error 108)
I20260618 22:39:59.758369 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 16510
I20260618 22:39:59.760916 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.760402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:34271 (local address 127.0.0.1:48516)
0618 22:39:59.760596 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:59.760713 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260618 22:39:59.761194 16300 catalog_manager.cc:4584] TS 710145f2a6454aba8553fac80080746e (127.5.213.193:34271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b99c19e9793e4c8abc0716dc17daba65: Network error: Client connection negotiation failed: client connection to 127.5.213.193:34271: connect: Connection refused (error 111)
Jun 18 22:39:59 dist-test-slave-lnj3 krb5kdc[16270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.194@KRBTEST.COM
I20260618 22:39:59.777179 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.759256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34478 (local address 127.5.213.194:36855)
0618 22:39:59.759439 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:59.759445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:59.759695 (+ 250us) server_negotiation.cc:407] Connection header received
0618 22:39:59.759781 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:59.759786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:59.759867 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:59.759964 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:59.761332 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.762114 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.763369 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.763755 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.763863 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:59.771453 (+ 7590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:59.771468 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:59.771484 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:59.771522 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:59.774312 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:59.775065 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:59.775076 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:59.775080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:59.775232 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:59.775624 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:59.775628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:59.775631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:59.776064 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:59.776347 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:59.776664 (+ 317us) server_negotiation.cc:299] Negotiation successful
0618 22:39:59.776872 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260618 22:39:59.777844 16633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:59.814994 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.814450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:34271 (local address 127.0.0.1:36206)
0618 22:39:59.814702 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:59.814829 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260618 22:39:59.816162 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.815785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:34271 (local address 127.0.0.1:36218)
0618 22:39:59.815923 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:59.816005 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260618 22:39:59.864569 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 16510
I20260618 22:39:59.876521 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.876090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:34271 (local address 127.0.0.1:36222)
0618 22:39:59.876301 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:59.876388 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260618 22:39:59.877480 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.877134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:34271 (local address 127.0.0.1:36238)
0618 22:39:59.877274 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:59.877342 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260618 22:39:59.883756 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 16643
Jun 18 22:39:59 dist-test-slave-lnj3 krb5kdc[16270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.195@KRBTEST.COM
I20260618 22:39:59.898814 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.884495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37580 (local address 127.5.213.195:34033)
0618 22:39:59.884673 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0618 22:39:59.884678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:39:59.884696 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:39:59.884763 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:39:59.884767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:39:59.884846 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:39:59.884937 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:39:59.886107 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.886973 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.888198 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:39:59.888648 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:39:59.888769 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:39:59.893207 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:39:59.893226 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:39:59.893241 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:39:59.893279 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:39:59.896089 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:59.896753 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:59.896761 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:59.896763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:59.896922 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:39:59.897270 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:39:59.897273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:39:59.897275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:39:59.897627 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:39:59.897903 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0618 22:39:59.898221 (+ 318us) server_negotiation.cc:299] Negotiation successful
0618 22:39:59.898427 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:39:59.899547 16766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:39:59.958149 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.957577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:34271 (local address 127.0.0.1:36252)
0618 22:39:59.957868 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:59.957981 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260618 22:39:59.959286 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:39:59.958817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:34271 (local address 127.0.0.1:36260)
0618 22:39:59.959014 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0618 22:39:59.959108 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260618 22:40:00.020561 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 16643
I20260618 22:40:00.035425 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 16283
I20260618 22:40:00.036509 16345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:40:00.142161 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:00.141614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:34271 (local address 127.0.0.1:36264)
0618 22:40:00.141808 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0618 22:40:00.141919 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:34271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260618 22:40:00.142367 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 16283
2026-06-18T22:40:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4135 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:40:00 dist-test-slave-lnj3 krb5kdc[16823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:40:00 dist-test-slave-lnj3 krb5kdc[16823](info): set up 2 sockets
Jun 18 22:40:00 dist-test-slave-lnj3 krb5kdc[16823](info): commencing operation
krb5kdc: starting...
W20260618 22:40:01.163285 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:40:01 dist-test-slave-lnj3 krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:40:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:40:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/rangeradmin_127.5.213.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/rangeradmin_127.5.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/rangerlookup_127.5.213.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/rangerlookup_127.5.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/HTTP_127.5.213.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/HTTP_127.5.213.212@KRBTEST.COM.keytab.
I20260618 22:40:01.458112 5975 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcnPxcP/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/postgres -l logfile start
2026-06-18 22:40:03.769 UTC [16853] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-18 22:40:03.769 UTC [16853] LOG: listening on IPv4 address "127.5.213.212", port 33685
2026-06-18 22:40:03.772 UTC [16853] LOG: listening on Unix socket "/tmp/.s.PGSQL.33685"
2026-06-18 22:40:03.792 UTC [16856] LOG: database system was shut down at 2026-06-18 22:40:02 UTC
2026-06-18 22:40:03.795 UTC [16853] LOG: database system is ready to accept connections
I20260618 22:40:04.694999 5975 mini_postgres.cc:96] Postgres bound to 33685
2026-06-18 22:40:04.710 UTC [16864] FATAL: database "slave" does not exist
127.5.213.212:33685 - accepting connections
I20260618 22:40:04.711241 5975 mini_ranger.cc:161] Starting Ranger...
I20260618 22:40:04.735596 5975 mini_ranger.cc:84] Created miniranger Postgres user
I20260618 22:40:05.194712 5975 mini_ranger.cc:87] Created ranger Postgres database
I20260618 22:40:05.194823 5975 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ranger-admin
2026-06-18 22:40:05,502 [I] DB FLAVOR :POSTGRES
2026-06-18 22:40:05,503 [I] --------- Verifying Ranger DB connection ---------
2026-06-18 22:40:05,503 [I] Checking connection..
2026-06-18 22:40:05,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-18 22:40:05,757 [I] Checking connection passed.
2026-06-18 22:40:05,757 [I] --------- Verifying version history table ---------
2026-06-18 22:40:05,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 22:40:06,000 [I] Table x_db_version_h does not exist in database ranger
2026-06-18 22:40:06,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 22:40:06,255 [I] Table x_db_version_h does not exist in database ranger
2026-06-18 22:40:06,255 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-18 22:40:06,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-18 22:40:06.509 UTC [16928] WARNING: there is no transaction in progress
2026-06-18 22:40:06,515 [I] create_dbversion_catalog.sql file imported successfully
2026-06-18 22:40:06,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 22:40:06,747 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-18 22:40:06,747 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-18 22:40:06,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-18 22:40:06,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-18 22:40:07,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lnj3.c.gcp-upstream.internal','N') ;"
2026-06-18 22:40:07,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-18 22:40:07,734 [I] Table x_portal_user does not exist in database ranger
2026-06-18 22:40:07,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-18 22:40:07,964 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-18 22:40:07,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-18 22:40:08,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-18 22:40:08,454 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-18 22:40:08,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-18 22:40:09.371 UTC [17036] WARNING: there is no transaction in progress
2026-06-18 22:40:09.386 UTC [17036] WARNING: there is no transaction in progress
2026-06-18 22:40:09.401 UTC [17036] WARNING: there is no transaction in progress
2026-06-18 22:40:09.416 UTC [17036] WARNING: there is no transaction in progress
2026-06-18 22:40:09.430 UTC [17036] WARNING: there is no transaction in progress
2026-06-18 22:40:09.448 UTC [17036] WARNING: there is no transaction in progress
2026-06-18 22:40:09.501 UTC [17036] WARNING: there is no transaction in progress
2026-06-18 22:40:09.507 UTC [17036] WARNING: there is no transaction in progress
2026-06-18 22:40:09.521 UTC [17036] WARNING: there is no transaction in progress
2026-06-18 22:40:09.530 UTC [17036] WARNING: there is no transaction in progress
2026-06-18 22:40:09.540 UTC [17036] WARNING: there is no transaction in progress
2026-06-18 22:40:09.553 UTC [17036] WARNING: there is no transaction in progress
2026-06-18 22:40:09.562 UTC [17036] WARNING: there is no transaction in progress
2026-06-18 22:40:09.568 UTC [17036] WARNING: there is no transaction in progress
2026-06-18 22:40:09.574 UTC [17036] WARNING: there is no transaction in progress
2026-06-18 22:40:09.580 UTC [17036] WARNING: there is no transaction in progress
2026-06-18 22:40:10,141 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-18 22:40:10,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-18 22:40:10,363 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-18 22:40:10,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-18 22:40:10,619 [I] Table x_portal_user already exists in database 'ranger'
2026-06-18 22:40:10,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-18 22:40:10,870 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-18 22:40:10,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-18 22:40:11,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-18 22:40:11,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lnj3.c.gcp-upstream.internal';"
2026-06-18 22:40:11,578 [I] CORE_DB_SCHEMA import status has been updated
2026-06-18 22:40:11,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-18 22:40:11,832 [I] DB_PATCHES have already been applied
I20260618 22:40:11.838567 5975 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcnPxcP/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260618 22:40:11.838689 5975 mini_ranger.cc:193] Using host: 127.5.213.212
I20260618 22:40:11.852237 5975 mini_ranger.cc:227] Ranger admin URL: http://127.5.213.212:39827
Jun 18, 2026 10:40:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 18, 2026 10:40:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 18, 2026 10:40:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ranger-admin/ews/webapp
Jun 18, 2026 10:40:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 18, 2026 10:40:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 18, 2026 10:40:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ranger-admin/ews/webapp].
Jun 18, 2026 10:40:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260618 22:40:12.884088 5975 mini_ranger.cc:160] Time spent starting Ranger: real 8.173s user 0.004s sys 0.038s
Jun 18, 2026 10:40:32 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 18, 2026 10:40:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 18, 2026 10:40:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 18, 2026 10:40:32 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 18, 2026 10:40:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 18, 2026 10:40:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 18, 2026 10:40:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 18, 2026 10:40:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 18, 2026 10:40:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 18, 2026 10:40:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 18, 2026 10:40:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 18, 2026 10:40:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 18, 2026 10:40:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 18, 2026 10:40:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 18, 2026 10:40:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 18, 2026 10:40:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 18, 2026 10:40:32 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 18, 2026 10:40:33 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260618 22:40:33.780308 5975 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/rangerkms_127.5.213.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/rangerkms_127.5.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.213.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.213.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.213.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/HTTP_127.5.213.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.213.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/HTTP_127.5.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 18 22:40:33 dist-test-slave-lnj3 krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822433, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260618 22:40:33.954003 5975 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260618 22:40:33.973306 5975 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260618 22:40:34.774353 5975 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260618 22:40:34.774509 5975 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ranger-kms
I20260618 22:40:34.774680 5975 mini_ranger_kms.cc:227] Using postgres at 127.5.213.212:33685
2026-06-18 22:40:35,233 [I] DB FLAVOR :POSTGRES
2026-06-18 22:40:35,233 [I] --------- Verifying Ranger DB connection ---------
2026-06-18 22:40:35,233 [I] Checking connection
2026-06-18 22:40:35,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-18 22:40:35,493 [I] connection success
2026-06-18 22:40:35,494 [I] --------- Verifying Ranger DB tables ---------
2026-06-18 22:40:35,494 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-18 22:40:35,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-18 22:40:35,811 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-18 22:40:35,811 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-18 22:40:35,811 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-18 22:40:35,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:33685/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-18 22:40:36,150 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260618 22:40:36.615233 5975 mini_ranger_kms.cc:310] Created kms service
I20260618 22:40:36.829316 5975 mini_ranger_kms.cc:326] Created kudu user
I20260618 22:40:36.874522 5975 mini_ranger_kms.cc:343] Created rangerkms user
Jun 18 22:40:36 dist-test-slave-lnj3 krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822433, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
I20260618 22:40:37.266352 5975 mini_ranger_kms.cc:384] Added ranger policy
I20260618 22:40:37.266697 5975 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260618 22:40:37.266757 5975 mini_ranger_kms.cc:240] Using host: 127.5.213.212
I20260618 22:40:37.286018 5975 mini_ranger_kms.cc:276] Ranger KMS PID: 17216
I20260618 22:40:37.286223 5975 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.213.212:39831
Jun 18, 2026 10:40:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 18, 2026 10:40:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 18, 2026 10:40:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 18, 2026 10:40:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 18, 2026 10:40:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ranger-kms/ews/webapp].
Jun 18, 2026 10:40:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260618 22:40:39.383399 5975 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.429s user 0.120s sys 0.361s
I20260618 22:40:39.383631 5975 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260618 22:40:39.383849 5975 mini_ranger_kms.cc:401] 127.5.213.212:39831/kms/v1/keys
Jun 18 22:40:39 dist-test-slave-lnj3 krb5kdc[16823](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781822439, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.213.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 18 22:40:39 dist-test-slave-lnj3 krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822439, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.213.212@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
Jun 18 22:40:40 dist-test-slave-lnj3 krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822439, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.213.212@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
Jun 18 22:40:42 dist-test-slave-lnj3 krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:40:42.387631 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:46213
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:46213
--ranger_config_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.213.212:39831/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:40:42.704919 17267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:40:42.705219 17267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:40:42.705338 17267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:40:42.712795 17267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:40:42.712916 17267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:40:42.712981 17267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:40:42.713063 17267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:40:42.713135 17267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:40:42.723721 17267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.213.212:39831/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:46213
--ranger_config_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:46213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:40:42.725061 17267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:40:42.726393 17267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:40:42.739840 17273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:40:42.739944 17272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:40:42.740553 17275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:40:42.741565 17267 server_base.cc:1034] running on GCE node
I20260618 22:40:42.742565 17267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:40:42.744488 17267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:40:42.745738 17267 hybrid_clock.cc:648] HybridClock initialized: now 1781822442745681 us; error 43 us; skew 500 ppm
I20260618 22:40:42.746145 17267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:40:42 dist-test-slave-lnj3 krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822442, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:40:42.753495 17267 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:40:42.755280 17267 webserver.cc:460] Webserver started at http://127.5.213.254:37677/ using document root <none> and password file <none>
I20260618 22:40:42.756129 17267 fs_manager.cc:359] Metadata directory not provided
I20260618 22:40:42.756266 17267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:40:42.756613 17267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:40:42 dist-test-slave-lnj3 krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822442, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
I20260618 22:40:42.813205 17267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "67b5c01bf77647e59ab5f4edf8867de3"
format_stamp: "Formatted at 2026-06-18 22:40:42 on dist-test-slave-lnj3"
server_key: "b41afde9043f28eb3aab00e1e326b241"
server_key_iv: "1d94edd3402397065758511025e348c0"
server_key_version: "kuduclusterkey@0"
I20260618 22:40:42.814268 17267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "67b5c01bf77647e59ab5f4edf8867de3"
format_stamp: "Formatted at 2026-06-18 22:40:42 on dist-test-slave-lnj3"
server_key: "b41afde9043f28eb3aab00e1e326b241"
server_key_iv: "1d94edd3402397065758511025e348c0"
server_key_version: "kuduclusterkey@0"
I20260618 22:40:42.821151 17267 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260618 22:40:42.840687 17282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:40:42.842206 17267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:40:42.842375 17267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "67b5c01bf77647e59ab5f4edf8867de3"
format_stamp: "Formatted at 2026-06-18 22:40:42 on dist-test-slave-lnj3"
server_key: "b41afde9043f28eb3aab00e1e326b241"
server_key_iv: "1d94edd3402397065758511025e348c0"
server_key_version: "kuduclusterkey@0"
I20260618 22:40:42.842584 17267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:40:42.864045 17267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:40:42.868010 17267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:40:42.868294 17267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:40:42.885866 17267 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:46213
I20260618 22:40:42.885890 17333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:46213 every 8 connection(s)
I20260618 22:40:42.887301 17267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:40:42.892155 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 17267
I20260618 22:40:42.892313 17334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:40:42.892334 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:40:42.892944 5975 external_mini_cluster.cc:1428] Setting key 9e30d7c32e1502c110812acbc90c986b
I20260618 22:40:42.906509 17334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3: Bootstrap starting.
I20260618 22:40:42.910567 17334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3: Neither blocks nor log segments found. Creating new log.
I20260618 22:40:42.911854 17334 log.cc:824] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3: Log is configured to *not* fsync() on all Append() calls
I20260618 22:40:42.914773 17334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3: No bootstrap required, opened a new log
Jun 18 22:40:42 dist-test-slave-lnj3 krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:40:42.923506 17334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b5c01bf77647e59ab5f4edf8867de3" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 46213 } }
I20260618 22:40:42.923884 17334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:40:42.923982 17334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b5c01bf77647e59ab5f4edf8867de3, State: Initialized, Role: FOLLOWER
I20260618 22:40:42.924602 17334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b5c01bf77647e59ab5f4edf8867de3" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 46213 } }
I20260618 22:40:42.924875 17334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:40:42.925050 17334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:40:42.925230 17334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:40:42.927989 17334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b5c01bf77647e59ab5f4edf8867de3" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 46213 } }
I20260618 22:40:42.928475 17334 leader_election.cc:302] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67b5c01bf77647e59ab5f4edf8867de3; no voters:
I20260618 22:40:42.929169 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:42.894930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49736 (local address 127.5.213.254:46213)
0618 22:40:42.907517 (+ 12587us) server_negotiation.cc:206] Beginning negotiation
0618 22:40:42.907529 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:40:42.907570 (+ 41us) server_negotiation.cc:407] Connection header received
0618 22:40:42.909278 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:40:42.909301 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:40:42.909694 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:40:42.910041 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:40:42.911351 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:42.912947 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:42.914749 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:42.915873 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:42.916059 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:42.920645 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:40:42.920656 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:40:42.920681 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:40:42.920731 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:40:42.924461 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:40:42.925289 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:40:42.925298 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:40:42.925307 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:40:42.925486 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:40:42.925907 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:40:42.925911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:40:42.925913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:40:42.926705 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:40:42.926996 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0618 22:40:42.927409 (+ 413us) server_negotiation.cc:299] Negotiation successful
0618 22:40:42.927802 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12364,"thread_start_us":12147,"threads_started":1}
I20260618 22:40:42.930940 17334 leader_election.cc:288] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:40:42.931035 17339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:40:42.937968 17339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3 [term 1 LEADER]: Becoming Leader. State: Replica: 67b5c01bf77647e59ab5f4edf8867de3, State: Running, Role: LEADER
I20260618 22:40:42.938660 17339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b5c01bf77647e59ab5f4edf8867de3" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 46213 } }
I20260618 22:40:42.939270 17334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:40:42.941712 17340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67b5c01bf77647e59ab5f4edf8867de3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b5c01bf77647e59ab5f4edf8867de3" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 46213 } } }
I20260618 22:40:42.942086 17340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3 [sys.catalog]: This master's current role is: LEADER
I20260618 22:40:42.942526 17341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67b5c01bf77647e59ab5f4edf8867de3. Latest consensus state: current_term: 1 leader_uuid: "67b5c01bf77647e59ab5f4edf8867de3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b5c01bf77647e59ab5f4edf8867de3" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 46213 } } }
I20260618 22:40:42.942849 17341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3 [sys.catalog]: This master's current role is: LEADER
I20260618 22:40:42.957795 17344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:40:42.963543 17344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:40:42.985214 17344 catalog_manager.cc:1347] Generated new cluster ID: 33e8520ef4c241d7bbe8091ba1734c97
I20260618 22:40:42.985347 17344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:40:43.011667 17334 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260618 22:40:43.017719 17344 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:40:43.019873 17344 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:40:43.037109 17344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3: Generated new TSK 0
I20260618 22:40:43.038516 17344 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
I20260618 22:40:43.822291 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.213.212:39831/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:46213
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:40:44 dist-test-slave-lnj3 krb5kdc[16823](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781822444, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260618 22:40:44.235618 17379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:40:44.236055 17379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:40:44.236268 17379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:40:44.248317 17379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:40:44.248545 17379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:40:44.248783 17379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:40:44.265079 17379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.213.212:39831/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:46213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:40:44.266997 17379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:40:44.268952 17379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:40:44.305045 17390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:40:44.306764 17392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:40:44.309001 17389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:40:44.309929 17379 server_base.cc:1034] running on GCE node
I20260618 22:40:44.310886 17379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:40:44.312304 17379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:40:44.313623 17379 hybrid_clock.cc:648] HybridClock initialized: now 1781822444313541 us; error 59 us; skew 500 ppm
I20260618 22:40:44.314047 17379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:40:44 dist-test-slave-lnj3 krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822444, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:40:44.318567 17379 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:40:44.324321 17379 webserver.cc:460] Webserver started at http://127.5.213.193:39047/ using document root <none> and password file <none>
I20260618 22:40:44.325340 17379 fs_manager.cc:359] Metadata directory not provided
I20260618 22:40:44.325533 17379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:40:44.326089 17379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:40:44 dist-test-slave-lnj3 krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822444, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
I20260618 22:40:44.344795 17379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "2164365c4e194996b8cb8636c308f49a"
format_stamp: "Formatted at 2026-06-18 22:40:44 on dist-test-slave-lnj3"
server_key: "6f2fbd08106d22d6f059d08ce6dbed14"
server_key_iv: "aaffd9f1b9996ea04326183bf3e8af9f"
server_key_version: "kuduclusterkey@0"
I20260618 22:40:44.345778 17379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "2164365c4e194996b8cb8636c308f49a"
format_stamp: "Formatted at 2026-06-18 22:40:44 on dist-test-slave-lnj3"
server_key: "6f2fbd08106d22d6f059d08ce6dbed14"
server_key_iv: "aaffd9f1b9996ea04326183bf3e8af9f"
server_key_version: "kuduclusterkey@0"
I20260618 22:40:44.354847 17379 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.011s sys 0.001s
I20260618 22:40:44.381657 17399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:40:44.383385 17379 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.000s
I20260618 22:40:44.383664 17379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "2164365c4e194996b8cb8636c308f49a"
format_stamp: "Formatted at 2026-06-18 22:40:44 on dist-test-slave-lnj3"
server_key: "6f2fbd08106d22d6f059d08ce6dbed14"
server_key_iv: "aaffd9f1b9996ea04326183bf3e8af9f"
server_key_version: "kuduclusterkey@0"
I20260618 22:40:44.384028 17379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:40:44.410619 17379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:40:44.416270 17379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:40:44.416639 17379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:40:44.421501 17379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:40:44.423623 17379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:40:44.423808 17379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260618 22:40:44.423962 17379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:40:44.424115 17379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:40:44.491611 17379 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:34025
I20260618 22:40:44.491671 17511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:34025 every 8 connection(s)
I20260618 22:40:44.493402 17379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:40:44.496428 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 17379
I20260618 22:40:44.496634 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:40:44.497383 5975 external_mini_cluster.cc:1428] Setting key 450597223a4708fcda73faa6ccf1c73e
Jun 18 22:40:44 dist-test-slave-lnj3 krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822444, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.213.194@KRBTEST.COM; defaulting to no policy
I20260618 22:40:44.531847 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:44.496665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:36055 (local address 127.5.213.254:46213)
0618 22:40:44.512643 (+ 15978us) server_negotiation.cc:206] Beginning negotiation
0618 22:40:44.512650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:40:44.512674 (+ 24us) server_negotiation.cc:407] Connection header received
0618 22:40:44.512750 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:40:44.512755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:40:44.512848 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:40:44.512948 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:40:44.515079 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:44.516195 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:44.518769 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:44.519407 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:44.519508 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:44.524123 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:40:44.524130 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:40:44.524136 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:40:44.524193 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:40:44.527612 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:40:44.528658 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:40:44.528664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:40:44.528668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:40:44.528827 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:40:44.529537 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:40:44.529544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:40:44.529547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:40:44.530005 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:40:44.530155 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0618 22:40:44.531336 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0618 22:40:44.531563 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11352,"thread_start_us":11217,"threads_started":1}
I20260618 22:40:44.533684 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:44.497053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:46213 (local address 127.5.213.193:36055)
0618 22:40:44.508834 (+ 11781us) negotiation.cc:107] Waiting for socket to connect
0618 22:40:44.508907 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0618 22:40:44.511305 (+ 2398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:40:44.513418 (+ 2113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:40:44.513432 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:40:44.514042 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:40:44.514874 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:40:44.514901 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:40:44.516415 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:40:44.516420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:40:44.518502 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:40:44.518533 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:40:44.519877 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:40:44.519884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:40:44.520177 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:44.521000 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:40:44.521032 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:40:44.523746 (+ 2714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:40:44.527882 (+ 4136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:40:44.527891 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:40:44.527906 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:40:44.528488 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:40:44.528959 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:40:44.528964 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:40:44.528968 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:40:44.529250 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:40:44.530271 (+ 1021us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:40:44.530280 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:40:44.530852 (+ 572us) client_negotiation.cc:769] Sending connection context
0618 22:40:44.532150 (+ 1298us) client_negotiation.cc:240] Negotiation successful
0618 22:40:44.532640 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11410,"thread_start_us":14585,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.194@KRBTEST.COM" created.
I20260618 22:40:44.536908 17512 heartbeater.cc:347] Connected to a master server at 127.5.213.254:46213
I20260618 22:40:44.537271 17512 heartbeater.cc:464] Registering TS with master...
I20260618 22:40:44.538194 17512 heartbeater.cc:511] Master 127.5.213.254:46213 requested a full tablet report, sending...
I20260618 22:40:44.541589 17299 ts_manager.cc:194] Registered new tserver with Master: 2164365c4e194996b8cb8636c308f49a (127.5.213.193:34025)
I20260618 22:40:44.543658 17299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:36055
Jun 18 22:40:44 dist-test-slave-lnj3 krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822444, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
I20260618 22:40:44.579181 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.213.212:39831/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:46213
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:40:44 dist-test-slave-lnj3 krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822444, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
W20260618 22:40:45.047269 17518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:40:45.047801 17518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:40:45.047991 17518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:40:45.059444 17518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:40:45.059674 17518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:40:45.059894 17518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:40:45.075516 17518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.213.212:39831/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:46213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:40:45.077633 17518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:40:45.079545 17518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:40:45.099438 17525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:40:45.099988 17526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:40:45.137882 17528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:40:45.138164 17518 server_base.cc:1034] running on GCE node
I20260618 22:40:45.139041 17518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:40:45.140673 17518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:40:45.142935 17518 hybrid_clock.cc:648] HybridClock initialized: now 1781822445142807 us; error 107 us; skew 500 ppm
I20260618 22:40:45.143370 17518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:40:45 dist-test-slave-lnj3 krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822445, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:40:45.147799 17518 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:40:45.151311 17518 webserver.cc:460] Webserver started at http://127.5.213.194:39565/ using document root <none> and password file <none>
I20260618 22:40:45.152073 17518 fs_manager.cc:359] Metadata directory not provided
I20260618 22:40:45.152179 17518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:40:45.152482 17518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:40:45 dist-test-slave-lnj3 krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822445, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
I20260618 22:40:45.167930 17518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "d0441eaf9db2499aa5bf18b6561eb863"
format_stamp: "Formatted at 2026-06-18 22:40:45 on dist-test-slave-lnj3"
server_key: "e2398f7e9f788a4c694b93ae572f6c75"
server_key_iv: "1d0bacff1f75eeb9976dab11c8de6253"
server_key_version: "kuduclusterkey@0"
I20260618 22:40:45.168794 17518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "d0441eaf9db2499aa5bf18b6561eb863"
format_stamp: "Formatted at 2026-06-18 22:40:45 on dist-test-slave-lnj3"
server_key: "e2398f7e9f788a4c694b93ae572f6c75"
server_key_iv: "1d0bacff1f75eeb9976dab11c8de6253"
server_key_version: "kuduclusterkey@0"
I20260618 22:40:45.174474 17518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260618 22:40:45.191179 17542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:40:45.193127 17518 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260618 22:40:45.193315 17518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "d0441eaf9db2499aa5bf18b6561eb863"
format_stamp: "Formatted at 2026-06-18 22:40:45 on dist-test-slave-lnj3"
server_key: "e2398f7e9f788a4c694b93ae572f6c75"
server_key_iv: "1d0bacff1f75eeb9976dab11c8de6253"
server_key_version: "kuduclusterkey@0"
I20260618 22:40:45.193586 17518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:40:45.245276 17518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:40:45.249555 17518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:40:45.249864 17518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:40:45.250828 17518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:40:45.252157 17518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:40:45.252233 17518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:40:45.252346 17518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:40:45.252396 17518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:40:45.283811 17518 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:45241
I20260618 22:40:45.283829 17654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:45241 every 8 connection(s)
I20260618 22:40:45.285100 17518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:40:45.294128 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 17518
I20260618 22:40:45.294319 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:40:45.294765 5975 external_mini_cluster.cc:1428] Setting key c813a554b552a0664361b9847d05465f
Jun 18 22:40:45 dist-test-slave-lnj3 krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822445, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:40:45.311314 17657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:45.287968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:57481 (local address 127.5.213.254:46213)
0618 22:40:45.288413 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0618 22:40:45.288419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:40:45.290719 (+ 2300us) server_negotiation.cc:407] Connection header received
0618 22:40:45.292112 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:40:45.292117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:40:45.292187 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:40:45.292287 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:40:45.294203 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.295369 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.297829 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.298435 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.298547 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:45.302983 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:40:45.302990 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:40:45.302995 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:40:45.303034 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:40:45.306421 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:40:45.307435 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:40:45.307441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:40:45.307445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:40:45.307602 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:40:45.308114 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:40:45.308119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:40:45.308122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:40:45.308465 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:40:45.308620 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0618 22:40:45.310671 (+ 2051us) server_negotiation.cc:299] Negotiation successful
0618 22:40:45.310938 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":219,"threads_started":1}
I20260618 22:40:45.311476 17658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:45.288600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:46213 (local address 127.5.213.194:57481)
0618 22:40:45.289393 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0618 22:40:45.289444 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 22:40:45.291800 (+ 2356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:40:45.292551 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:40:45.292565 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:40:45.293151 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:40:45.293980 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:40:45.294007 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.295576 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.295582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:40:45.297622 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:40:45.297667 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.298965 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.298974 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:40:45.299289 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:45.300049 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:40:45.300080 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:40:45.302725 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:40:45.306657 (+ 3932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:40:45.306669 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:40:45.306690 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:40:45.307285 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:40:45.307731 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:40:45.307736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:40:45.307739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:40:45.307990 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:40:45.308694 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:40:45.308702 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:40:45.309134 (+ 432us) client_negotiation.cc:769] Sending connection context
0618 22:40:45.309509 (+ 375us) client_negotiation.cc:240] Negotiation successful
0618 22:40:45.310070 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":301,"threads_started":1}
I20260618 22:40:45.313131 17655 heartbeater.cc:347] Connected to a master server at 127.5.213.254:46213
I20260618 22:40:45.313442 17655 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.213.195@KRBTEST.COM; defaulting to no policy
I20260618 22:40:45.314245 17655 heartbeater.cc:511] Master 127.5.213.254:46213 requested a full tablet report, sending...
I20260618 22:40:45.315793 17299 ts_manager.cc:194] Registered new tserver with Master: d0441eaf9db2499aa5bf18b6561eb863 (127.5.213.194:45241)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.195@KRBTEST.COM" created.
I20260618 22:40:45.316735 17299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:57481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
I20260618 22:40:45.344193 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.213.212:39831/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:46213
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:40:45.547809 17512 heartbeater.cc:503] Master 127.5.213.254:46213 was elected leader, sending a full tablet report...
W20260618 22:40:45.663098 17661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:40:45.663383 17661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:40:45.663511 17661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:40:45.670588 17661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:40:45.670725 17661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:40:45.670871 17661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:40:45.680560 17661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.213.212:39831/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:46213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:40:45.681840 17661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:40:45.683100 17661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:40:45.695492 17666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:40:45.695452 17667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:40:45.695526 17669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:40:45.696606 17661 server_base.cc:1034] running on GCE node
I20260618 22:40:45.697165 17661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:40:45.698065 17661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:40:45.699390 17661 hybrid_clock.cc:648] HybridClock initialized: now 1781822445699332 us; error 57 us; skew 500 ppm
I20260618 22:40:45.699712 17661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:40:45 dist-test-slave-lnj3 krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822445, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:40:45.702679 17661 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:40:45.704012 17661 webserver.cc:460] Webserver started at http://127.5.213.195:45485/ using document root <none> and password file <none>
I20260618 22:40:45.704664 17661 fs_manager.cc:359] Metadata directory not provided
I20260618 22:40:45.704777 17661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:40:45.705060 17661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:40:45 dist-test-slave-lnj3 krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822445, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
I20260618 22:40:45.721155 17661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "57217ec761e94b85a968ea48899ba18d"
format_stamp: "Formatted at 2026-06-18 22:40:45 on dist-test-slave-lnj3"
server_key: "a6112332441c749bef9563413605fa10"
server_key_iv: "b8c63bc948b8300a82a36ff7794d1f66"
server_key_version: "kuduclusterkey@0"
I20260618 22:40:45.722235 17661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "57217ec761e94b85a968ea48899ba18d"
format_stamp: "Formatted at 2026-06-18 22:40:45 on dist-test-slave-lnj3"
server_key: "a6112332441c749bef9563413605fa10"
server_key_iv: "b8c63bc948b8300a82a36ff7794d1f66"
server_key_version: "kuduclusterkey@0"
I20260618 22:40:45.728096 17661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260618 22:40:45.744813 17676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:40:45.746515 17661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:40:45.746768 17661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "57217ec761e94b85a968ea48899ba18d"
format_stamp: "Formatted at 2026-06-18 22:40:45 on dist-test-slave-lnj3"
server_key: "a6112332441c749bef9563413605fa10"
server_key_iv: "b8c63bc948b8300a82a36ff7794d1f66"
server_key_version: "kuduclusterkey@0"
I20260618 22:40:45.747146 17661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:40:45.761909 17661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:40:45.765941 17661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:40:45.766252 17661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:40:45.767275 17661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:40:45.768652 17661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:40:45.768735 17661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:40:45.768862 17661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:40:45.768923 17661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:40:45.800377 17661 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:34221
I20260618 22:40:45.800410 17788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:34221 every 8 connection(s)
I20260618 22:40:45.801800 17661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:40:45.807647 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 17661
I20260618 22:40:45.807816 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:40:45.808367 5975 external_mini_cluster.cc:1428] Setting key 8c3b09186e365eb1c5bf496b1c2fd03a
Jun 18 22:40:45 dist-test-slave-lnj3 krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822445, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:40:45.821034 17657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:45.804356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:40691 (local address 127.5.213.254:46213)
0618 22:40:45.804528 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0618 22:40:45.804533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:40:45.805654 (+ 1121us) server_negotiation.cc:407] Connection header received
0618 22:40:45.807435 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:40:45.807440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:40:45.807533 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:40:45.807617 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:40:45.809625 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.810510 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.812417 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.812817 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.812906 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:45.815969 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:40:45.815974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:40:45.815978 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:40:45.816013 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:40:45.818327 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:40:45.819036 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:40:45.819041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:40:45.819043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:40:45.819194 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:40:45.819591 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:40:45.819594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:40:45.819596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:40:45.819879 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:40:45.819980 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 22:40:45.820643 (+ 663us) server_negotiation.cc:299] Negotiation successful
0618 22:40:45.820809 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 22:40:45.822029 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:45.804859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:46213 (local address 127.5.213.195:40691)
0618 22:40:45.805506 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0618 22:40:45.805548 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:40:45.807151 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:40:45.807903 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:40:45.807914 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:40:45.808468 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:40:45.809222 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:40:45.809248 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.810785 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.810789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:40:45.812284 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:40:45.812296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.812987 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.812992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:40:45.813195 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:45.813791 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:40:45.813812 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:40:45.815750 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:40:45.818439 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:40:45.818445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:40:45.818458 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:40:45.818910 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:40:45.819293 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:40:45.819296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:40:45.819299 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:40:45.819484 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:40:45.819976 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:40:45.819981 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:40:45.820284 (+ 303us) client_negotiation.cc:769] Sending connection context
0618 22:40:45.820501 (+ 217us) client_negotiation.cc:240] Negotiation successful
0618 22:40:45.820868 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":254,"threads_started":1}
I20260618 22:40:45.823437 17789 heartbeater.cc:347] Connected to a master server at 127.5.213.254:46213
I20260618 22:40:45.823776 17789 heartbeater.cc:464] Registering TS with master...
I20260618 22:40:45.824456 17789 heartbeater.cc:511] Master 127.5.213.254:46213 requested a full tablet report, sending...
I20260618 22:40:45.825891 17299 ts_manager.cc:194] Registered new tserver with Master: 57217ec761e94b85a968ea48899ba18d (127.5.213.195:34221)
I20260618 22:40:45.826759 17299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:40691
I20260618 22:40:45.835842 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:40:45.852123 17657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:45.840010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49738 (local address 127.5.213.254:46213)
0618 22:40:45.840200 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0618 22:40:45.840205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:40:45.840621 (+ 416us) server_negotiation.cc:407] Connection header received
0618 22:40:45.840774 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:40:45.840778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:40:45.840853 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:40:45.840935 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:40:45.842278 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.843078 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.844355 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.844967 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.845103 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:45.847558 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:40:45.847563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:40:45.847566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:40:45.847598 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:40:45.850028 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:40:45.850571 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:40:45.850575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:40:45.850578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:40:45.850729 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:40:45.851045 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:40:45.851047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:40:45.851049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:40:45.851285 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:40:45.851535 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0618 22:40:45.851713 (+ 178us) server_negotiation.cc:299] Negotiation successful
0618 22:40:45.851871 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 22:40:45.855727 17299 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:40:45.862902 17299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:40:45.884874 17804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:45.875828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35710 (local address 127.5.213.195:34221)
0618 22:40:45.876795 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0618 22:40:45.876804 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:40:45.876836 (+ 32us) server_negotiation.cc:407] Connection header received
0618 22:40:45.876924 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:40:45.876931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:40:45.877195 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:40:45.878218 (+ 1023us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:40:45.878525 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.880051 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.882776 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.883691 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.883807 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:45.884048 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0618 22:40:45.884369 (+ 321us) server_negotiation.cc:299] Negotiation successful
0618 22:40:45.884530 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":632,"threads_started":1}
I20260618 22:40:45.884914 17803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:45.875447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:34221 (local address 127.0.0.1:35710)
0618 22:40:45.875951 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0618 22:40:45.875972 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:40:45.876073 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:40:45.877450 (+ 1377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:40:45.877459 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:40:45.877493 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:40:45.877910 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:40:45.877922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.880201 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.880206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:40:45.882557 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:40:45.882569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.883888 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.883894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:40:45.884154 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:45.884203 (+ 49us) client_negotiation.cc:769] Sending connection context
0618 22:40:45.884311 (+ 108us) client_negotiation.cc:240] Negotiation successful
0618 22:40:45.884506 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":45,"thread_start_us":298,"threads_started":1}
I20260618 22:40:45.887208 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:45.876497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41872 (local address 127.5.213.194:45241)
0618 22:40:45.877098 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0618 22:40:45.877107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:40:45.879208 (+ 2101us) server_negotiation.cc:407] Connection header received
0618 22:40:45.879461 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:40:45.879466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:40:45.879641 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:40:45.879800 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:40:45.880621 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.881870 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.884396 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.885799 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.885907 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:45.886087 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0618 22:40:45.886750 (+ 663us) server_negotiation.cc:299] Negotiation successful
0618 22:40:45.886922 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":318,"threads_started":1}
I20260618 22:40:45.887275 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:45.876067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:45241 (local address 127.0.0.1:41872)
0618 22:40:45.879100 (+ 3033us) negotiation.cc:107] Waiting for socket to connect
0618 22:40:45.879118 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:40:45.879302 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:40:45.879799 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:40:45.879804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:40:45.879826 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:40:45.880201 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:40:45.880215 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.882019 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.882024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:40:45.884221 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:40:45.884236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.886298 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.886304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:40:45.886523 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:45.886557 (+ 34us) client_negotiation.cc:769] Sending connection context
0618 22:40:45.886703 (+ 146us) client_negotiation.cc:240] Negotiation successful
0618 22:40:45.886854 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2917,"mutex_wait_us":53,"thread_start_us":2773,"threads_started":1}
I20260618 22:40:45.887655 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:45.874293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:34025 (local address 127.0.0.1:41972)
0618 22:40:45.875088 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0618 22:40:45.875139 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 22:40:45.875324 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:40:45.880014 (+ 4690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:40:45.880023 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:40:45.880050 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:40:45.880581 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:40:45.880593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.882701 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.882707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:40:45.884745 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:40:45.884758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.886894 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.886900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:40:45.887069 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:45.887099 (+ 30us) client_negotiation.cc:769] Sending connection context
0618 22:40:45.887164 (+ 65us) client_negotiation.cc:240] Negotiation successful
0618 22:40:45.887310 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":48,"thread_start_us":365,"threads_started":1}
I20260618 22:40:45.888662 17802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:45.874514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.5.213.193:34025)
0618 22:40:45.879402 (+ 4888us) server_negotiation.cc:206] Beginning negotiation
0618 22:40:45.879410 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:40:45.879436 (+ 26us) server_negotiation.cc:407] Connection header received
0618 22:40:45.879529 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:40:45.879536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:40:45.879761 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:40:45.880344 (+ 583us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:40:45.880751 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.882539 (+ 1788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.884928 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:45.886679 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:45.887865 (+ 1186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:45.888101 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0618 22:40:45.888173 (+ 72us) server_negotiation.cc:299] Negotiation successful
0618 22:40:45.888335 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4733,"thread_start_us":4594,"threads_started":1}
I20260618 22:40:45.891443 17590 tablet_service.cc:1467] Processing CreateTablet for tablet 5d3b908aba7c448a97f4a662503213ac (DEFAULT_TABLE table=test-table [id=bffdbf3673714c3fb46a8dd6df986d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:40:45.892170 17447 tablet_service.cc:1467] Processing CreateTablet for tablet 5d3b908aba7c448a97f4a662503213ac (DEFAULT_TABLE table=test-table [id=bffdbf3673714c3fb46a8dd6df986d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:40:45.892189 17724 tablet_service.cc:1467] Processing CreateTablet for tablet 5d3b908aba7c448a97f4a662503213ac (DEFAULT_TABLE table=test-table [id=bffdbf3673714c3fb46a8dd6df986d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:40:45.893617 17724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d3b908aba7c448a97f4a662503213ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:40:45.893616 17447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d3b908aba7c448a97f4a662503213ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:40:45.895241 17590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d3b908aba7c448a97f4a662503213ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:40:45.910195 17807 tablet_bootstrap.cc:492] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a: Bootstrap starting.
I20260618 22:40:45.912000 17809 tablet_bootstrap.cc:492] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863: Bootstrap starting.
I20260618 22:40:45.913765 17807 tablet_bootstrap.cc:654] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a: Neither blocks nor log segments found. Creating new log.
I20260618 22:40:45.914598 17809 tablet_bootstrap.cc:654] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863: Neither blocks nor log segments found. Creating new log.
I20260618 22:40:45.914811 17807 log.cc:824] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:40:45.915215 17808 tablet_bootstrap.cc:492] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d: Bootstrap starting.
I20260618 22:40:45.916046 17809 log.cc:824] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863: Log is configured to *not* fsync() on all Append() calls
I20260618 22:40:45.919605 17808 tablet_bootstrap.cc:654] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d: Neither blocks nor log segments found. Creating new log.
I20260618 22:40:45.920827 17808 log.cc:824] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d: Log is configured to *not* fsync() on all Append() calls
I20260618 22:40:45.921258 17807 tablet_bootstrap.cc:492] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a: No bootstrap required, opened a new log
I20260618 22:40:45.921355 17809 tablet_bootstrap.cc:492] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863: No bootstrap required, opened a new log
I20260618 22:40:45.921550 17807 ts_tablet_manager.cc:1397] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260618 22:40:45.921692 17809 ts_tablet_manager.cc:1397] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260618 22:40:45.923964 17808 tablet_bootstrap.cc:492] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d: No bootstrap required, opened a new log
I20260618 22:40:45.924222 17808 ts_tablet_manager.cc:1397] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260618 22:40:45.929098 17807 raft_consensus.cc:348] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2164365c4e194996b8cb8636c308f49a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34025 } } peers { permanent_uuid: "57217ec761e94b85a968ea48899ba18d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34221 } } peers { permanent_uuid: "d0441eaf9db2499aa5bf18b6561eb863" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 45241 } }
I20260618 22:40:45.929464 17807 raft_consensus.cc:374] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:40:45.929585 17807 raft_consensus.cc:729] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2164365c4e194996b8cb8636c308f49a, State: Initialized, Role: FOLLOWER
I20260618 22:40:45.930194 17807 consensus_queue.cc:260] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2164365c4e194996b8cb8636c308f49a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34025 } } peers { permanent_uuid: "57217ec761e94b85a968ea48899ba18d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34221 } } peers { permanent_uuid: "d0441eaf9db2499aa5bf18b6561eb863" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 45241 } }
I20260618 22:40:45.931679 17809 raft_consensus.cc:348] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2164365c4e194996b8cb8636c308f49a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34025 } } peers { permanent_uuid: "57217ec761e94b85a968ea48899ba18d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34221 } } peers { permanent_uuid: "d0441eaf9db2499aa5bf18b6561eb863" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 45241 } }
I20260618 22:40:45.932111 17809 raft_consensus.cc:374] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:40:45.932267 17809 raft_consensus.cc:729] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0441eaf9db2499aa5bf18b6561eb863, State: Initialized, Role: FOLLOWER
I20260618 22:40:45.933043 17807 ts_tablet_manager.cc:1428] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a: Time spent starting tablet: real 0.011s user 0.009s sys 0.002s
I20260618 22:40:45.932979 17809 consensus_queue.cc:260] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2164365c4e194996b8cb8636c308f49a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34025 } } peers { permanent_uuid: "57217ec761e94b85a968ea48899ba18d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34221 } } peers { permanent_uuid: "d0441eaf9db2499aa5bf18b6561eb863" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 45241 } }
I20260618 22:40:45.934188 17655 heartbeater.cc:503] Master 127.5.213.254:46213 was elected leader, sending a full tablet report...
I20260618 22:40:45.934900 17809 ts_tablet_manager.cc:1428] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260618 22:40:45.934583 17808 raft_consensus.cc:348] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2164365c4e194996b8cb8636c308f49a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34025 } } peers { permanent_uuid: "57217ec761e94b85a968ea48899ba18d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34221 } } peers { permanent_uuid: "d0441eaf9db2499aa5bf18b6561eb863" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 45241 } }
I20260618 22:40:45.935029 17808 raft_consensus.cc:374] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:40:45.935148 17808 raft_consensus.cc:729] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57217ec761e94b85a968ea48899ba18d, State: Initialized, Role: FOLLOWER
I20260618 22:40:45.935793 17808 consensus_queue.cc:260] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2164365c4e194996b8cb8636c308f49a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34025 } } peers { permanent_uuid: "57217ec761e94b85a968ea48899ba18d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34221 } } peers { permanent_uuid: "d0441eaf9db2499aa5bf18b6561eb863" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 45241 } }
I20260618 22:40:45.936865 17789 heartbeater.cc:503] Master 127.5.213.254:46213 was elected leader, sending a full tablet report...
I20260618 22:40:45.937462 17808 ts_tablet_manager.cc:1428] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260618 22:40:45.997424 17513 tablet.cc:2369] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:40:46.037535 17656 tablet.cc:2369] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:40:46.053396 17790 tablet.cc:2369] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:40:46.057595 17815 raft_consensus.cc:482] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:40:46.057842 17815 raft_consensus.cc:504] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2164365c4e194996b8cb8636c308f49a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34025 } } peers { permanent_uuid: "57217ec761e94b85a968ea48899ba18d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34221 } } peers { permanent_uuid: "d0441eaf9db2499aa5bf18b6561eb863" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 45241 } }
I20260618 22:40:46.059594 17815 leader_election.cc:288] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2164365c4e194996b8cb8636c308f49a (127.5.213.193:34025), d0441eaf9db2499aa5bf18b6561eb863 (127.5.213.194:45241)
I20260618 22:40:46.066015 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:46.059810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:34025 (local address 127.5.213.195:44473)
0618 22:40:46.059968 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0618 22:40:46.059989 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:40:46.060110 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:40:46.060669 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:40:46.060674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:40:46.060701 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:40:46.060883 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:40:46.060895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:40:46.062128 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:40:46.062132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:40:46.064071 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:40:46.064085 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:40:46.065427 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:40:46.065431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:40:46.065558 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:46.065582 (+ 24us) client_negotiation.cc:769] Sending connection context
0618 22:40:46.065649 (+ 67us) client_negotiation.cc:240] Negotiation successful
0618 22:40:46.065741 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":37}
I20260618 22:40:46.066075 17802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:46.060042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:44473 (local address 127.5.213.193:34025)
0618 22:40:46.060220 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0618 22:40:46.060235 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:40:46.060255 (+ 20us) server_negotiation.cc:407] Connection header received
0618 22:40:46.060318 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:40:46.060321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:40:46.060386 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:40:46.060491 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:40:46.061035 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:46.061996 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:46.064242 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:46.065282 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:46.065436 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:46.065517 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 22:40:46.065725 (+ 208us) server_negotiation.cc:299] Negotiation successful
0618 22:40:46.065862 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:40:46.066601 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:46.060468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:49465 (local address 127.5.213.194:45241)
0618 22:40:46.060652 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0618 22:40:46.060657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:40:46.060910 (+ 253us) server_negotiation.cc:407] Connection header received
0618 22:40:46.061059 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:40:46.061064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:40:46.061147 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:40:46.061251 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:40:46.061683 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:46.062477 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:46.064630 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:46.065725 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:46.065864 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:46.065924 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0618 22:40:46.066259 (+ 335us) server_negotiation.cc:299] Negotiation successful
0618 22:40:46.066376 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260618 22:40:46.067214 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:46.060193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:45241 (local address 127.5.213.195:49465)
0618 22:40:46.060809 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0618 22:40:46.060830 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:40:46.060942 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:40:46.061269 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:40:46.061271 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:40:46.061290 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:40:46.061554 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:40:46.061561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:40:46.062621 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:40:46.062652 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:40:46.064493 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:40:46.064503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:40:46.065909 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:40:46.065914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:40:46.066073 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:46.066105 (+ 32us) client_negotiation.cc:769] Sending connection context
0618 22:40:46.066801 (+ 696us) client_negotiation.cc:240] Negotiation successful
0618 22:40:46.066926 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":24,"thread_start_us":289,"threads_started":1}
I20260618 22:40:46.067750 17467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3b908aba7c448a97f4a662503213ac" candidate_uuid: "57217ec761e94b85a968ea48899ba18d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2164365c4e194996b8cb8636c308f49a" is_pre_election: true
I20260618 22:40:46.068097 17610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3b908aba7c448a97f4a662503213ac" candidate_uuid: "57217ec761e94b85a968ea48899ba18d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0441eaf9db2499aa5bf18b6561eb863" is_pre_election: true
I20260618 22:40:46.068307 17467 raft_consensus.cc:2454] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57217ec761e94b85a968ea48899ba18d in term 0.
I20260618 22:40:46.068536 17610 raft_consensus.cc:2454] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57217ec761e94b85a968ea48899ba18d in term 0.
I20260618 22:40:46.068977 17680 leader_election.cc:302] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2164365c4e194996b8cb8636c308f49a, 57217ec761e94b85a968ea48899ba18d; no voters:
I20260618 22:40:46.069305 17815 raft_consensus.cc:2781] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:40:46.069444 17815 raft_consensus.cc:482] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:40:46.069553 17815 raft_consensus.cc:3037] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:40:46.071653 17815 raft_consensus.cc:504] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2164365c4e194996b8cb8636c308f49a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34025 } } peers { permanent_uuid: "57217ec761e94b85a968ea48899ba18d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34221 } } peers { permanent_uuid: "d0441eaf9db2499aa5bf18b6561eb863" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 45241 } }
I20260618 22:40:46.072187 17815 leader_election.cc:288] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [CANDIDATE]: Term 1 election: Requested vote from peers 2164365c4e194996b8cb8636c308f49a (127.5.213.193:34025), d0441eaf9db2499aa5bf18b6561eb863 (127.5.213.194:45241)
I20260618 22:40:46.072631 17467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3b908aba7c448a97f4a662503213ac" candidate_uuid: "57217ec761e94b85a968ea48899ba18d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2164365c4e194996b8cb8636c308f49a"
I20260618 22:40:46.072711 17610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3b908aba7c448a97f4a662503213ac" candidate_uuid: "57217ec761e94b85a968ea48899ba18d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0441eaf9db2499aa5bf18b6561eb863"
I20260618 22:40:46.072803 17467 raft_consensus.cc:3037] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:40:46.072898 17610 raft_consensus.cc:3037] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:40:46.075023 17467 raft_consensus.cc:2454] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57217ec761e94b85a968ea48899ba18d in term 1.
I20260618 22:40:46.075052 17610 raft_consensus.cc:2454] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57217ec761e94b85a968ea48899ba18d in term 1.
I20260618 22:40:46.075399 17680 leader_election.cc:302] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57217ec761e94b85a968ea48899ba18d, d0441eaf9db2499aa5bf18b6561eb863; no voters:
I20260618 22:40:46.075665 17815 raft_consensus.cc:2781] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:40:46.076118 17815 raft_consensus.cc:686] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [term 1 LEADER]: Becoming Leader. State: Replica: 57217ec761e94b85a968ea48899ba18d, State: Running, Role: LEADER
I20260618 22:40:46.076587 17815 consensus_queue.cc:237] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2164365c4e194996b8cb8636c308f49a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34025 } } peers { permanent_uuid: "57217ec761e94b85a968ea48899ba18d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34221 } } peers { permanent_uuid: "d0441eaf9db2499aa5bf18b6561eb863" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 45241 } }
I20260618 22:40:46.080943 17298 catalog_manager.cc:5495] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d reported cstate change: term changed from 0 to 1, leader changed from <none> to 57217ec761e94b85a968ea48899ba18d (127.5.213.195). New cstate: current_term: 1 leader_uuid: "57217ec761e94b85a968ea48899ba18d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2164365c4e194996b8cb8636c308f49a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57217ec761e94b85a968ea48899ba18d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0441eaf9db2499aa5bf18b6561eb863" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 45241 } health_report { overall_health: UNKNOWN } } }
I20260618 22:40:46.125819 17804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:46.120450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35720 (local address 127.5.213.195:34221)
0618 22:40:46.120667 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0618 22:40:46.120672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:40:46.120688 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:40:46.120820 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:40:46.120825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:40:46.120911 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:40:46.121056 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:40:46.121443 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:46.122188 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:46.123693 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:46.124086 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:46.124196 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:46.124697 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:40:46.125233 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:40:46.125384 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0618 22:40:46.125483 (+ 99us) server_negotiation.cc:299] Negotiation successful
0618 22:40:46.125570 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260618 22:40:46.133949 17467 raft_consensus.cc:1261] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a [term 1 FOLLOWER]: Refusing update from remote peer 57217ec761e94b85a968ea48899ba18d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:40:46.133949 17610 raft_consensus.cc:1261] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863 [term 1 FOLLOWER]: Refusing update from remote peer 57217ec761e94b85a968ea48899ba18d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:40:46.134986 17817 consensus_queue.cc:1035] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0441eaf9db2499aa5bf18b6561eb863" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 45241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:40:46.135426 17815 consensus_queue.cc:1035] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2164365c4e194996b8cb8636c308f49a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 34025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:40:46.148741 17822 mvcc.cc:205] Tried to move back new op lower bound from 7298344739354415104 to 7298344739133218816. Current Snapshot: MvccSnapshot[applied={T|T < 7298344739354415104}]
I20260618 22:40:46.154536 17821 mvcc.cc:205] Tried to move back new op lower bound from 7298344739354415104 to 7298344739133218816. Current Snapshot: MvccSnapshot[applied={T|T < 7298344739354415104}]
I20260618 22:40:46.164142 17298 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:40:46.164404 17298 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:40:46.167971 17298 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67b5c01bf77647e59ab5f4edf8867de3: Sending DeleteTablet for 3 replicas of tablet 5d3b908aba7c448a97f4a662503213ac
I20260618 22:40:46.169261 17590 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d3b908aba7c448a97f4a662503213ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:40:46 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:41872
I20260618 22:40:46.169266 17724 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d3b908aba7c448a97f4a662503213ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:40:46 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:35710
I20260618 22:40:46.169323 17447 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d3b908aba7c448a97f4a662503213ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:40:46 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:41972
I20260618 22:40:46.171912 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 17379
I20260618 22:40:46.184592 17831 tablet_replica.cc:330] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a: stopping tablet replica
I20260618 22:40:46.185125 17831 raft_consensus.cc:2229] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:40:46.185741 17831 raft_consensus.cc:2258] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:40:46.191165 17829 tablet_replica.cc:330] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863: stopping tablet replica
I20260618 22:40:46.191639 17829 raft_consensus.cc:2229] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:40:46.192152 17829 raft_consensus.cc:2258] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:40:46.192153 17831 ts_tablet_manager.cc:1905] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:40:46.194221 17829 ts_tablet_manager.cc:1905] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:40:46.197741 17830 tablet_replica.cc:330] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d: stopping tablet replica
Jun 18 22:40:46 dist-test-slave-lnj3 krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:40:46.198268 17830 raft_consensus.cc:2229] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:40:46.199793 17831 ts_tablet_manager.cc:1918] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:40:46.200029 17831 log.cc:1196] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/5d3b908aba7c448a97f4a662503213ac
I20260618 22:40:46.200222 17830 raft_consensus.cc:2258] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:40:46.200589 17831 ts_tablet_manager.cc:1939] T 5d3b908aba7c448a97f4a662503213ac P 2164365c4e194996b8cb8636c308f49a: Deleting consensus metadata
I20260618 22:40:46.201037 17829 ts_tablet_manager.cc:1918] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:40:46.201241 17829 log.cc:1196] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/5d3b908aba7c448a97f4a662503213ac
I20260618 22:40:46.201752 17829 ts_tablet_manager.cc:1939] T 5d3b908aba7c448a97f4a662503213ac P d0441eaf9db2499aa5bf18b6561eb863: Deleting consensus metadata
I20260618 22:40:46.202832 17286 catalog_manager.cc:4841] TS 2164365c4e194996b8cb8636c308f49a (127.5.213.193:34025): tablet 5d3b908aba7c448a97f4a662503213ac (table test-table [id=bffdbf3673714c3fb46a8dd6df986d69]) successfully deleted
I20260618 22:40:46.203734 17286 catalog_manager.cc:4841] TS d0441eaf9db2499aa5bf18b6561eb863 (127.5.213.194:45241): tablet 5d3b908aba7c448a97f4a662503213ac (table test-table [id=bffdbf3673714c3fb46a8dd6df986d69]) successfully deleted
I20260618 22:40:46.206712 17830 ts_tablet_manager.cc:1905] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:40:46.208209 17802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:46.173556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.5.213.193:34025)
0618 22:40:46.173783 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0618 22:40:46.173790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:40:46.185163 (+ 11373us) server_negotiation.cc:407] Connection header received
0618 22:40:46.185468 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:40:46.185475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:40:46.185567 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:40:46.185647 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:40:46.187601 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:46.188800 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:46.192796 (+ 3996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:46.193526 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:46.193650 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:46.200596 (+ 6946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:40:46.200614 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:40:46.200633 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:40:46.200678 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:40:46.204608 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:40:46.205829 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:40:46.205839 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:40:46.205842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:40:46.206005 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:40:46.206438 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:40:46.206442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:40:46.206445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:40:46.206943 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:40:46.207230 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0618 22:40:46.207586 (+ 356us) server_negotiation.cc:299] Negotiation successful
0618 22:40:46.207844 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260618 22:40:46.208935 17507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:40:46.213609 17830 ts_tablet_manager.cc:1918] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:40:46.213785 17830 log.cc:1196] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/5d3b908aba7c448a97f4a662503213ac
I20260618 22:40:46.214277 17830 ts_tablet_manager.cc:1939] T 5d3b908aba7c448a97f4a662503213ac P 57217ec761e94b85a968ea48899ba18d: Deleting consensus metadata
I20260618 22:40:46.216197 17286 catalog_manager.cc:4841] TS 57217ec761e94b85a968ea48899ba18d (127.5.213.195:34221): tablet 5d3b908aba7c448a97f4a662503213ac (table test-table [id=bffdbf3673714c3fb46a8dd6df986d69]) successfully deleted
I20260618 22:40:46.309681 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 17379
I20260618 22:40:46.323680 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 17518
Jun 18 22:40:46 dist-test-slave-lnj3 krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.194@KRBTEST.COM
I20260618 22:40:46.340448 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:46.324458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41884 (local address 127.5.213.194:45241)
0618 22:40:46.324610 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0618 22:40:46.324614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:40:46.324632 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:40:46.324783 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:40:46.324787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:40:46.324847 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:40:46.324925 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:40:46.326543 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:46.327592 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:46.329422 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:46.329830 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:46.329992 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:46.334916 (+ 4924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:40:46.334931 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:40:46.334947 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:40:46.334994 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:40:46.337653 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:40:46.338331 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:40:46.338339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:40:46.338341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:40:46.338470 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:40:46.338862 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:40:46.338868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:40:46.338871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:40:46.339282 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:40:46.339587 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0618 22:40:46.339885 (+ 298us) server_negotiation.cc:299] Negotiation successful
0618 22:40:46.340094 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:40:46.341264 17650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:40:46.430614 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 17518
I20260618 22:40:46.449855 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 17661
Jun 18 22:40:46 dist-test-slave-lnj3 krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.195@KRBTEST.COM
I20260618 22:40:46.468462 17804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:40:46.450797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35726 (local address 127.5.213.195:34221)
0618 22:40:46.450969 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0618 22:40:46.450973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:40:46.451009 (+ 36us) server_negotiation.cc:407] Connection header received
0618 22:40:46.451244 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:40:46.451249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:40:46.451314 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:40:46.451406 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:40:46.453177 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:46.454020 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:46.455964 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:40:46.456445 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:40:46.456560 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:40:46.462368 (+ 5808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:40:46.462387 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:40:46.462404 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:40:46.462461 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:40:46.465435 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:40:46.466276 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:40:46.466288 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:40:46.466291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:40:46.466457 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:40:46.466892 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:40:46.466897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:40:46.466899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:40:46.467264 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:40:46.467589 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0618 22:40:46.467888 (+ 299us) server_negotiation.cc:299] Negotiation successful
0618 22:40:46.468132 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260618 22:40:46.469132 17784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:40:46.596047 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 17661
I20260618 22:40:46.616882 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 17267
I20260618 22:40:46.618211 17329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:40:46.733115 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 17267
I20260618 22:40:46.751003 5975 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260618 22:40:47.086958 5975 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260618 22:40:47.087178 5975 mini_ranger.cc:66] Stopping Ranger...
I20260618 22:40:47.422832 5975 mini_ranger.cc:68] Stopped Ranger
2026-06-18 22:40:47.423 UTC [16853] LOG: received smart shutdown request
2026-06-18 22:40:47.426 UTC [16853] LOG: background worker "logical replication launcher" (PID 16862) exited with exit code 1
2026-06-18 22:40:47.426 UTC [16857] LOG: shutting down
2026-06-18 22:40:47.577 UTC [16853] LOG: database system is shut down
2026-06-18T22:40:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47553 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:40:47 dist-test-slave-lnj3 krb5kdc[17845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:40:47 dist-test-slave-lnj3 krb5kdc[17845](info): set up 2 sockets
Jun 18 22:40:47 dist-test-slave-lnj3 krb5kdc[17845](info): commencing operation
krb5kdc: starting...
W20260618 22:40:48.718986 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:40:48 dist-test-slave-lnj3 krb5kdc[17845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:40:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:40:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/rangeradmin_127.5.213.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/rangeradmin_127.5.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/rangerlookup_127.5.213.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/rangerlookup_127.5.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/HTTP_127.5.213.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/HTTP_127.5.213.212@KRBTEST.COM.keytab.
I20260618 22:40:49.033517 5975 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcnPxcP/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/postgres -l logfile start
2026-06-18 22:40:51.578 UTC [17875] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-18 22:40:51.578 UTC [17875] LOG: listening on IPv4 address "127.5.213.212", port 60529
2026-06-18 22:40:51.582 UTC [17875] LOG: listening on Unix socket "/tmp/.s.PGSQL.60529"
2026-06-18 22:40:51.604 UTC [17878] LOG: database system was shut down at 2026-06-18 22:40:50 UTC
2026-06-18 22:40:51.607 UTC [17875] LOG: database system is ready to accept connections
I20260618 22:40:52.498500 5975 mini_postgres.cc:96] Postgres bound to 60529
2026-06-18 22:40:52.518 UTC [17886] FATAL: database "slave" does not exist
127.5.213.212:60529 - accepting connections
I20260618 22:40:52.519517 5975 mini_ranger.cc:161] Starting Ranger...
I20260618 22:40:52.544306 5975 mini_ranger.cc:84] Created miniranger Postgres user
I20260618 22:40:52.998405 5975 mini_ranger.cc:87] Created ranger Postgres database
I20260618 22:40:52.998502 5975 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ranger-admin
2026-06-18 22:40:53,268 [I] DB FLAVOR :POSTGRES
2026-06-18 22:40:53,268 [I] --------- Verifying Ranger DB connection ---------
2026-06-18 22:40:53,268 [I] Checking connection..
2026-06-18 22:40:53,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-18 22:40:53,511 [I] Checking connection passed.
2026-06-18 22:40:53,511 [I] --------- Verifying version history table ---------
2026-06-18 22:40:53,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 22:40:53,740 [I] Table x_db_version_h does not exist in database ranger
2026-06-18 22:40:53,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 22:40:53,971 [I] Table x_db_version_h does not exist in database ranger
2026-06-18 22:40:53,971 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-18 22:40:53,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-18 22:40:54.202 UTC [17950] WARNING: there is no transaction in progress
2026-06-18 22:40:54,208 [I] create_dbversion_catalog.sql file imported successfully
2026-06-18 22:40:54,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 22:40:54,470 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-18 22:40:54,471 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-18 22:40:54,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-18 22:40:54,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-18 22:40:54,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lnj3.c.gcp-upstream.internal','N') ;"
2026-06-18 22:40:55,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-18 22:40:55,432 [I] Table x_portal_user does not exist in database ranger
2026-06-18 22:40:55,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-18 22:40:55,697 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-18 22:40:55,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-18 22:40:55,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-18 22:40:56,185 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-18 22:40:56,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-18 22:40:57.105 UTC [18058] WARNING: there is no transaction in progress
2026-06-18 22:40:57.120 UTC [18058] WARNING: there is no transaction in progress
2026-06-18 22:40:57.136 UTC [18058] WARNING: there is no transaction in progress
2026-06-18 22:40:57.151 UTC [18058] WARNING: there is no transaction in progress
2026-06-18 22:40:57.167 UTC [18058] WARNING: there is no transaction in progress
2026-06-18 22:40:57.184 UTC [18058] WARNING: there is no transaction in progress
2026-06-18 22:40:57.239 UTC [18058] WARNING: there is no transaction in progress
2026-06-18 22:40:57.245 UTC [18058] WARNING: there is no transaction in progress
2026-06-18 22:40:57.258 UTC [18058] WARNING: there is no transaction in progress
2026-06-18 22:40:57.268 UTC [18058] WARNING: there is no transaction in progress
2026-06-18 22:40:57.278 UTC [18058] WARNING: there is no transaction in progress
2026-06-18 22:40:57.292 UTC [18058] WARNING: there is no transaction in progress
2026-06-18 22:40:57.302 UTC [18058] WARNING: there is no transaction in progress
2026-06-18 22:40:57.308 UTC [18058] WARNING: there is no transaction in progress
2026-06-18 22:40:57.314 UTC [18058] WARNING: there is no transaction in progress
2026-06-18 22:40:57.321 UTC [18058] WARNING: there is no transaction in progress
2026-06-18 22:40:57,912 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-18 22:40:57,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-18 22:40:58,149 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-18 22:40:58,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-18 22:40:58,398 [I] Table x_portal_user already exists in database 'ranger'
2026-06-18 22:40:58,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-18 22:40:58,662 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-18 22:40:58,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-18 22:40:58,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-18 22:40:59,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lnj3.c.gcp-upstream.internal';"
2026-06-18 22:40:59,405 [I] CORE_DB_SCHEMA import status has been updated
2026-06-18 22:40:59,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-18 22:40:59,644 [I] DB_PATCHES have already been applied
I20260618 22:40:59.650065 5975 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcnPxcP/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260618 22:40:59.650146 5975 mini_ranger.cc:193] Using host: 127.5.213.212
I20260618 22:40:59.662714 5975 mini_ranger.cc:227] Ranger admin URL: http://127.5.213.212:48373
Jun 18, 2026 10:40:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 18, 2026 10:40:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 18, 2026 10:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ranger-admin/ews/webapp
Jun 18, 2026 10:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 18, 2026 10:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 18, 2026 10:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ranger-admin/ews/webapp].
Jun 18, 2026 10:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260618 22:41:00.663983 5975 mini_ranger.cc:160] Time spent starting Ranger: real 8.144s user 0.002s sys 0.033s
Jun 18, 2026 10:41:19 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 18, 2026 10:41:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 18, 2026 10:41:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 18, 2026 10:41:19 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 18, 2026 10:41:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 18, 2026 10:41:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 18, 2026 10:41:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 18, 2026 10:41:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 18, 2026 10:41:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 18, 2026 10:41:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 18, 2026 10:41:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 18, 2026 10:41:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 18, 2026 10:41:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 18, 2026 10:41:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 18, 2026 10:41:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 18, 2026 10:41:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 18, 2026 10:41:19 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 18, 2026 10:41:20 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260618 22:41:21.422679 5975 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/rangerkms_127.5.213.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/rangerkms_127.5.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.213.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.213.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.213.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/HTTP_127.5.213.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.213.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/HTTP_127.5.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 18 22:41:21 dist-test-slave-lnj3 krb5kdc[17845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822481, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260618 22:41:21.601413 5975 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260618 22:41:21.623865 5975 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260618 22:41:22.473057 5975 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260618 22:41:22.473153 5975 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ranger-kms
I20260618 22:41:22.473192 5975 mini_ranger_kms.cc:227] Using postgres at 127.5.213.212:60529
2026-06-18 22:41:22,811 [I] DB FLAVOR :POSTGRES
2026-06-18 22:41:22,811 [I] --------- Verifying Ranger DB connection ---------
2026-06-18 22:41:22,812 [I] Checking connection
2026-06-18 22:41:22,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-18 22:41:23,042 [I] connection success
2026-06-18 22:41:23,043 [I] --------- Verifying Ranger DB tables ---------
2026-06-18 22:41:23,043 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-18 22:41:23,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-18 22:41:23,279 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-18 22:41:23,279 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-18 22:41:23,279 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-18 22:41:23,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.213.212:60529/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-18 22:41:23,530 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260618 22:41:24.012502 5975 mini_ranger_kms.cc:310] Created kms service
I20260618 22:41:24.234529 5975 mini_ranger_kms.cc:326] Created kudu user
I20260618 22:41:24.275394 5975 mini_ranger_kms.cc:343] Created rangerkms user
Jun 18 22:41:24 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822481, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
I20260618 22:41:24.658818 5975 mini_ranger_kms.cc:384] Added ranger policy
I20260618 22:41:24.659123 5975 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcnPxcP/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcnPxcP/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260618 22:41:24.659183 5975 mini_ranger_kms.cc:240] Using host: 127.5.213.212
I20260618 22:41:24.675042 5975 mini_ranger_kms.cc:276] Ranger KMS PID: 18238
I20260618 22:41:24.675172 5975 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.213.212:57679
Jun 18, 2026 10:41:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 18, 2026 10:41:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 18, 2026 10:41:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 18, 2026 10:41:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 18, 2026 10:41:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ranger-kms/ews/webapp].
Jun 18, 2026 10:41:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260618 22:41:26.722147 5975 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.121s user 0.058s sys 0.297s
I20260618 22:41:26.722393 5975 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260618 22:41:26.722553 5975 mini_ranger_kms.cc:401] 127.5.213.212:57679/kms/v1/keys
Jun 18 22:41:26 dist-test-slave-lnj3 krb5kdc[17845](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781822486, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.213.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 18 22:41:27 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822486, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.213.212@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
Jun 18 22:41:27 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822486, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.213.212@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
Jun 18 22:41:29 dist-test-slave-lnj3 krb5kdc[17845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:41:29.885936 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:39477
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:39477
--ranger_config_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.213.212:57679/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:41:30.212921 18289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:30.213209 18289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:30.213335 18289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:30.220492 18289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:41:30.220609 18289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:30.220687 18289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:30.220764 18289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:41:30.220824 18289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:41:30.231007 18289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.213.212:57679/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:39477
--ranger_config_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:39477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:30.232316 18289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:30.233613 18289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:30.245103 18295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:30.245334 18289 server_base.cc:1034] running on GCE node
W20260618 22:41:30.245160 18294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:30.245134 18297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:30.246414 18289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:30.247936 18289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:30.249135 18289 hybrid_clock.cc:648] HybridClock initialized: now 1781822490249093 us; error 35 us; skew 500 ppm
I20260618 22:41:30.249468 18289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:30 dist-test-slave-lnj3 krb5kdc[17845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822490, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:30.252485 18289 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:41:30.253892 18289 webserver.cc:460] Webserver started at http://127.5.213.254:35399/ using document root <none> and password file <none>
I20260618 22:41:30.254536 18289 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:30.254674 18289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:41:30.254966 18289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:41:30 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822490, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
I20260618 22:41:30.312896 18289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "27e98d09d96f45278d9368b95f54004a"
format_stamp: "Formatted at 2026-06-18 22:41:30 on dist-test-slave-lnj3"
server_key: "6a3f5f1236af4013171032a03a85fc16"
server_key_iv: "09d76cee703b6700ebd68b06e1d9d4f5"
server_key_version: "kuduclusterkey@0"
I20260618 22:41:30.314039 18289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "27e98d09d96f45278d9368b95f54004a"
format_stamp: "Formatted at 2026-06-18 22:41:30 on dist-test-slave-lnj3"
server_key: "6a3f5f1236af4013171032a03a85fc16"
server_key_iv: "09d76cee703b6700ebd68b06e1d9d4f5"
server_key_version: "kuduclusterkey@0"
I20260618 22:41:30.321231 18289 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260618 22:41:30.341841 18304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:30.343760 18289 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260618 22:41:30.343962 18289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "27e98d09d96f45278d9368b95f54004a"
format_stamp: "Formatted at 2026-06-18 22:41:30 on dist-test-slave-lnj3"
server_key: "6a3f5f1236af4013171032a03a85fc16"
server_key_iv: "09d76cee703b6700ebd68b06e1d9d4f5"
server_key_version: "kuduclusterkey@0"
I20260618 22:41:30.344228 18289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:30.394084 18289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:30.399341 18289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:30.399695 18289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:30.418179 18289 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:39477
I20260618 22:41:30.418207 18355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:39477 every 8 connection(s)
I20260618 22:41:30.419520 18289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:41:30.420888 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 18289
I20260618 22:41:30.421059 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:41:30.421711 5975 external_mini_cluster.cc:1428] Setting key 401575381c856a393d3a188a10afd63c
I20260618 22:41:30.423566 18356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 18 22:41:30 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:41:30.436338 18356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a: Bootstrap starting.
I20260618 22:41:30.439589 18356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a: Neither blocks nor log segments found. Creating new log.
I20260618 22:41:30.440635 18356 log.cc:824] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:41:30.443419 18356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a: No bootstrap required, opened a new log
I20260618 22:41:30.443632 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:30.423629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51684 (local address 127.5.213.254:39477)
0618 22:41:30.424370 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:30.424381 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:30.424420 (+ 39us) server_negotiation.cc:407] Connection header received
0618 22:41:30.425973 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:30.425993 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:30.426376 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:30.426743 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:30.428297 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:30.429634 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:30.431513 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:30.432182 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:30.432316 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:30.436593 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:30.436601 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:30.436616 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:30.436657 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:30.439862 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:30.440502 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:30.440508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:30.440514 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:30.440667 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:30.441007 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:30.441010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:30.441012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:30.441538 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:30.441841 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:30.442235 (+ 394us) server_negotiation.cc:299] Negotiation successful
0618 22:41:30.442600 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":331,"threads_started":1}
I20260618 22:41:30.450836 18356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e98d09d96f45278d9368b95f54004a" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 39477 } }
I20260618 22:41:30.451265 18356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:41:30.451360 18356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27e98d09d96f45278d9368b95f54004a, State: Initialized, Role: FOLLOWER
I20260618 22:41:30.451926 18356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e98d09d96f45278d9368b95f54004a" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 39477 } }
I20260618 22:41:30.452131 18356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:41:30.452252 18356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:41:30.452389 18356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:41:30.454452 18356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e98d09d96f45278d9368b95f54004a" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 39477 } }
I20260618 22:41:30.454958 18356 leader_election.cc:302] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27e98d09d96f45278d9368b95f54004a; no voters:
I20260618 22:41:30.455544 18356 leader_election.cc:288] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:41:30.455681 18361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:41:30.456389 18361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [term 1 LEADER]: Becoming Leader. State: Replica: 27e98d09d96f45278d9368b95f54004a, State: Running, Role: LEADER
I20260618 22:41:30.456933 18361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e98d09d96f45278d9368b95f54004a" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 39477 } }
I20260618 22:41:30.457664 18356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:41:30.459610 18362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27e98d09d96f45278d9368b95f54004a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e98d09d96f45278d9368b95f54004a" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 39477 } } }
I20260618 22:41:30.460455 18362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [sys.catalog]: This master's current role is: LEADER
I20260618 22:41:30.460237 18363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27e98d09d96f45278d9368b95f54004a. Latest consensus state: current_term: 1 leader_uuid: "27e98d09d96f45278d9368b95f54004a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e98d09d96f45278d9368b95f54004a" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 39477 } } }
I20260618 22:41:30.461509 18363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [sys.catalog]: This master's current role is: LEADER
I20260618 22:41:30.462875 18370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:41:30.467167 18370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:41:30.467926 18356 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260618 22:41:30.489735 18370 catalog_manager.cc:1347] Generated new cluster ID: 48be31faff394372bf3dc32253a22323
I20260618 22:41:30.489908 18370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:41:30.512964 18370 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:41:30.514413 18370 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:41:30.521508 18370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a: Generated new TSK 0
I20260618 22:41:30.522612 18370 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
I20260618 22:41:31.238430 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.213.212:57679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:39477
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:41:31 dist-test-slave-lnj3 krb5kdc[17845](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781822491, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260618 22:41:31.671315 18401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:31.671693 18401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:31.671895 18401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:31.683540 18401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:31.683735 18401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:31.683944 18401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:41:31.699558 18401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.213.212:57679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:39477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:31.701337 18401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:31.702963 18401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:31.722054 18412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:31.722054 18411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:31.722733 18414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:31.723032 18401 server_base.cc:1034] running on GCE node
I20260618 22:41:31.723671 18401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:31.724665 18401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:31.725906 18401 hybrid_clock.cc:648] HybridClock initialized: now 1781822491725841 us; error 53 us; skew 500 ppm
I20260618 22:41:31.726231 18401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:31 dist-test-slave-lnj3 krb5kdc[17845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822491, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:31.729393 18401 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:41:31.730988 18401 webserver.cc:460] Webserver started at http://127.5.213.193:36091/ using document root <none> and password file <none>
I20260618 22:41:31.731676 18401 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:31.731781 18401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:41:31.732074 18401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:41:31 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822491, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
I20260618 22:41:31.759538 18401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "63a51bf4b84f46ec9871b7a150ecca78"
format_stamp: "Formatted at 2026-06-18 22:41:31 on dist-test-slave-lnj3"
server_key: "1a87b944e303a4322f129ac67d07a5f3"
server_key_iv: "87909d5f785bba1e3b260834c2350352"
server_key_version: "kuduclusterkey@0"
I20260618 22:41:31.760848 18401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "63a51bf4b84f46ec9871b7a150ecca78"
format_stamp: "Formatted at 2026-06-18 22:41:31 on dist-test-slave-lnj3"
server_key: "1a87b944e303a4322f129ac67d07a5f3"
server_key_iv: "87909d5f785bba1e3b260834c2350352"
server_key_version: "kuduclusterkey@0"
I20260618 22:41:31.778329 18401 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.003s sys 0.011s
I20260618 22:41:31.810353 18421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:31.816330 18401 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.002s sys 0.009s
I20260618 22:41:31.816668 18401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "63a51bf4b84f46ec9871b7a150ecca78"
format_stamp: "Formatted at 2026-06-18 22:41:31 on dist-test-slave-lnj3"
server_key: "1a87b944e303a4322f129ac67d07a5f3"
server_key_iv: "87909d5f785bba1e3b260834c2350352"
server_key_version: "kuduclusterkey@0"
I20260618 22:41:31.817065 18401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:31.878590 18401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:31.883796 18401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:31.884117 18401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:31.885317 18401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:41:31.887115 18401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:41:31.887210 18401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:31.887391 18401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:41:31.887452 18401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:41:31 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822491, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
I20260618 22:41:32.002451 18401 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:39597
I20260618 22:41:32.002514 18533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:39597 every 8 connection(s)
I20260618 22:41:32.004120 18401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:41:32.008466 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 18401
I20260618 22:41:32.008671 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:41:32.009482 5975 external_mini_cluster.cc:1428] Setting key 30ad936ec9298e180538b0ec572d8fd9
WARNING: no policy specified for kudu/127.5.213.194@KRBTEST.COM; defaulting to no policy
Jun 18 22:41:32 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822491, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.194@KRBTEST.COM" created.
I20260618 22:41:32.048269 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:32.007103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:48685 (local address 127.5.213.254:39477)
0618 22:41:32.023093 (+ 15990us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:32.023100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:32.023128 (+ 28us) server_negotiation.cc:407] Connection header received
0618 22:41:32.026277 (+ 3149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:32.026282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:32.026387 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:32.026486 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:32.030272 (+ 3786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:32.031425 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:32.033874 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:32.034641 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:32.034748 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:32.038956 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:32.038962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:32.038967 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:32.039006 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:32.044677 (+ 5671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:32.045536 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:32.045541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:32.045545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:32.045701 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:32.046214 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:32.046218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:32.046221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:32.046554 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:32.046690 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:32.047649 (+ 959us) server_negotiation.cc:299] Negotiation successful
0618 22:41:32.047879 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15832,"thread_start_us":23502,"threads_started":1}
I20260618 22:41:32.050762 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:32.007700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:39477 (local address 127.5.213.193:48685)
0618 22:41:32.010132 (+ 2432us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:32.010195 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:32.025666 (+ 15471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:32.028551 (+ 2885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:32.028564 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:32.029177 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:32.030047 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:32.030076 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:32.031643 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:32.031648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:32.033690 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:32.033706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:32.035116 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:32.035123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:32.035447 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:32.036230 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:32.036271 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:32.038732 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:32.044844 (+ 6112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:32.044853 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:32.044871 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:32.045407 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:32.045836 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:32.045840 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:32.045843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:32.046102 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:32.046785 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:32.046792 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:32.047221 (+ 429us) client_negotiation.cc:769] Sending connection context
0618 22:41:32.048773 (+ 1552us) client_negotiation.cc:240] Negotiation successful
0618 22:41:32.049301 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1997,"thread_start_us":2363,"threads_started":1}
I20260618 22:41:32.052815 18534 heartbeater.cc:347] Connected to a master server at 127.5.213.254:39477
I20260618 22:41:32.053117 18534 heartbeater.cc:464] Registering TS with master...
I20260618 22:41:32.053920 18534 heartbeater.cc:511] Master 127.5.213.254:39477 requested a full tablet report, sending...
I20260618 22:41:32.056387 18321 ts_manager.cc:194] Registered new tserver with Master: 63a51bf4b84f46ec9871b7a150ecca78 (127.5.213.193:39597)
I20260618 22:41:32.058370 18321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:48685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
I20260618 22:41:32.071827 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.213.212:57679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:39477
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:41:32 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822491, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
W20260618 22:41:32.467515 18540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:32.468007 18540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:32.468199 18540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:32.479935 18540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:32.480082 18540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:32.480309 18540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:41:32.495952 18540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.213.212:57679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:39477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:32.497619 18540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:32.499346 18540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:41:32.513674 18540 server_base.cc:1034] running on GCE node
W20260618 22:41:32.513607 18554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:32.513733 18555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:32.513690 18557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:32.514693 18540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:32.515618 18540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:32.516844 18540 hybrid_clock.cc:648] HybridClock initialized: now 1781822492516775 us; error 56 us; skew 500 ppm
I20260618 22:41:32.517199 18540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:32 dist-test-slave-lnj3 krb5kdc[17845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822492, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:32.520323 18540 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:41:32.521765 18540 webserver.cc:460] Webserver started at http://127.5.213.194:39691/ using document root <none> and password file <none>
I20260618 22:41:32.522418 18540 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:32.522528 18540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:41:32.523025 18540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:41:32 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822492, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
I20260618 22:41:32.537392 18540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "f2c8a8d3a3c544e78ae045c76335f58b"
format_stamp: "Formatted at 2026-06-18 22:41:32 on dist-test-slave-lnj3"
server_key: "09329398b6374dd7a37107901038c85b"
server_key_iv: "2b2b132f0c5c7b2fa310e4ec967b9679"
server_key_version: "kuduclusterkey@0"
I20260618 22:41:32.538167 18540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "f2c8a8d3a3c544e78ae045c76335f58b"
format_stamp: "Formatted at 2026-06-18 22:41:32 on dist-test-slave-lnj3"
server_key: "09329398b6374dd7a37107901038c85b"
server_key_iv: "2b2b132f0c5c7b2fa310e4ec967b9679"
server_key_version: "kuduclusterkey@0"
I20260618 22:41:32.543529 18540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260618 22:41:32.560719 18564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:32.562800 18540 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260618 22:41:32.563010 18540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "f2c8a8d3a3c544e78ae045c76335f58b"
format_stamp: "Formatted at 2026-06-18 22:41:32 on dist-test-slave-lnj3"
server_key: "09329398b6374dd7a37107901038c85b"
server_key_iv: "2b2b132f0c5c7b2fa310e4ec967b9679"
server_key_version: "kuduclusterkey@0"
I20260618 22:41:32.563275 18540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:32.578222 18540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:32.582289 18540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:32.582566 18540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:32.583562 18540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:41:32.584987 18540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:41:32.585058 18540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:32.585162 18540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:41:32.585211 18540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:32.621556 18540 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:46375
I20260618 22:41:32.621626 18676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:46375 every 8 connection(s)
I20260618 22:41:32.622874 18540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:41:32.627640 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 18540
I20260618 22:41:32.627823 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:41:32.628265 5975 external_mini_cluster.cc:1428] Setting key 2318b9b29c1d67fd895b2dba3a12e271
Jun 18 22:41:32 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822492, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:41:32.646852 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:32.625689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:37713 (local address 127.5.213.254:39477)
0618 22:41:32.626179 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:32.626185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:32.627192 (+ 1007us) server_negotiation.cc:407] Connection header received
0618 22:41:32.629622 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:32.629628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:32.629723 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:32.629825 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:32.632077 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:32.633172 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:32.636930 (+ 3758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:32.637390 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:32.637495 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:32.641085 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:32.641090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:32.641093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:32.641127 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:32.643647 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:32.644513 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:32.644517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:32.644519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:32.644635 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:32.645069 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:32.645073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:32.645075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:32.645356 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:32.645484 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:32.646193 (+ 709us) server_negotiation.cc:299] Negotiation successful
0618 22:41:32.646452 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":228,"threads_started":1}
I20260618 22:41:32.647789 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:32.626199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:39477 (local address 127.5.213.194:37713)
0618 22:41:32.627014 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:32.627072 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:32.629320 (+ 2248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:32.630403 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:32.630415 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:32.631042 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:32.631880 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:32.631907 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:32.634749 (+ 2842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:32.634754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:32.636746 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:32.636764 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:32.637625 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:32.637633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:32.637942 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:32.638650 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:32.638683 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:32.640844 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:32.643779 (+ 2935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:32.643789 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:32.643809 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:32.644409 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:32.644756 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:32.644761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:32.644765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:32.644950 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:32.645483 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:32.645492 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:32.645836 (+ 344us) client_negotiation.cc:769] Sending connection context
0618 22:41:32.646074 (+ 238us) client_negotiation.cc:240] Negotiation successful
0618 22:41:32.646602 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":311,"threads_started":1}
WARNING: no policy specified for kudu/127.5.213.195@KRBTEST.COM; defaulting to no policy
I20260618 22:41:32.649200 18677 heartbeater.cc:347] Connected to a master server at 127.5.213.254:39477
I20260618 22:41:32.649493 18677 heartbeater.cc:464] Registering TS with master...
I20260618 22:41:32.650274 18677 heartbeater.cc:511] Master 127.5.213.254:39477 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.195@KRBTEST.COM" created.
I20260618 22:41:32.651901 18321 ts_manager.cc:194] Registered new tserver with Master: f2c8a8d3a3c544e78ae045c76335f58b (127.5.213.194:46375)
I20260618 22:41:32.652967 18321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:37713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
I20260618 22:41:32.677211 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.213.212:57679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:39477
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:41:32.985801 18683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:32.986078 18683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:32.986208 18683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:32.993397 18683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:32.993525 18683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:32.993675 18683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:41:33.003543 18683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.213.212:57679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:39477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:33.004751 18683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:33.005942 18683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:33.017985 18688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:33.018016 18691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:33.017954 18689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:33.018726 18683 server_base.cc:1034] running on GCE node
I20260618 22:41:33.019276 18683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:33.020172 18683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:33.021365 18683 hybrid_clock.cc:648] HybridClock initialized: now 1781822493021316 us; error 45 us; skew 500 ppm
I20260618 22:41:33.021656 18683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:33 dist-test-slave-lnj3 krb5kdc[17845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822493, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:33.024649 18683 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:41:33.026026 18683 webserver.cc:460] Webserver started at http://127.5.213.195:35733/ using document root <none> and password file <none>
I20260618 22:41:33.026715 18683 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:33.026836 18683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:41:33.027114 18683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:41:33 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822493, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
I20260618 22:41:33.040938 18683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "2bf9c3df528b4758a96f459e463d4e85"
format_stamp: "Formatted at 2026-06-18 22:41:33 on dist-test-slave-lnj3"
server_key: "4256c11411218877b3f4d0789a063f5e"
server_key_iv: "e5c5a4444b54133304b81e32c5b9e5d7"
server_key_version: "kuduclusterkey@0"
I20260618 22:41:33.041723 18683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "2bf9c3df528b4758a96f459e463d4e85"
format_stamp: "Formatted at 2026-06-18 22:41:33 on dist-test-slave-lnj3"
server_key: "4256c11411218877b3f4d0789a063f5e"
server_key_iv: "e5c5a4444b54133304b81e32c5b9e5d7"
server_key_version: "kuduclusterkey@0"
I20260618 22:41:33.047302 18683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260618 22:41:33.062213 18534 heartbeater.cc:503] Master 127.5.213.254:39477 was elected leader, sending a full tablet report...
I20260618 22:41:33.064764 18698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:33.066305 18683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:41:33.066454 18683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "2bf9c3df528b4758a96f459e463d4e85"
format_stamp: "Formatted at 2026-06-18 22:41:33 on dist-test-slave-lnj3"
server_key: "4256c11411218877b3f4d0789a063f5e"
server_key_iv: "e5c5a4444b54133304b81e32c5b9e5d7"
server_key_version: "kuduclusterkey@0"
I20260618 22:41:33.066790 18683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:33.091747 18683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:33.095139 18683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:33.095405 18683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:33.096323 18683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:41:33.097572 18683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:41:33.097640 18683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:33.097749 18683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:41:33.097803 18683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:33.126494 18683 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:44477
I20260618 22:41:33.126504 18810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:44477 every 8 connection(s)
I20260618 22:41:33.127758 18683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:41:33.136785 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 18683
I20260618 22:41:33.136936 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:41:33.137498 5975 external_mini_cluster.cc:1428] Setting key 687ceb3e3b0ba25d99defa52b02c1574
Jun 18 22:41:33 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822493, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:41:33.146282 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:33.130148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:57539 (local address 127.5.213.254:39477)
0618 22:41:33.130317 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:33.130321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:33.131499 (+ 1178us) server_negotiation.cc:407] Connection header received
0618 22:41:33.133238 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:33.133252 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:33.133354 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:33.133441 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:33.134799 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:33.135490 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:33.136985 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:33.137593 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:33.137725 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:33.141117 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:33.141121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:33.141125 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:33.141156 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:33.143498 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:33.144171 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:33.144175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:33.144178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:33.144306 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:33.144681 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:33.144684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:33.144686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:33.144954 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:33.145084 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:33.145730 (+ 646us) server_negotiation.cc:299] Negotiation successful
0618 22:41:33.145957 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:41:33.147208 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:33.130696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:39477 (local address 127.5.213.195:57539)
0618 22:41:33.131349 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:33.131385 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:33.132976 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:33.133671 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:33.133680 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:33.134067 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:33.134613 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:33.134656 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:33.135625 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:33.135629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:33.136831 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:33.136843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:33.137798 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:33.137804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:33.138003 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:33.138641 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:33.138679 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:33.140898 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:33.143621 (+ 2723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:33.143629 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:33.143643 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:33.144051 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:33.144405 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:33.144409 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:33.144413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:33.144574 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:33.145073 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:33.145080 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:33.145388 (+ 308us) client_negotiation.cc:769] Sending connection context
0618 22:41:33.145613 (+ 225us) client_negotiation.cc:240] Negotiation successful
0618 22:41:33.146099 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":249,"threads_started":1}
I20260618 22:41:33.148425 18811 heartbeater.cc:347] Connected to a master server at 127.5.213.254:39477
I20260618 22:41:33.148762 18811 heartbeater.cc:464] Registering TS with master...
I20260618 22:41:33.149468 18811 heartbeater.cc:511] Master 127.5.213.254:39477 requested a full tablet report, sending...
I20260618 22:41:33.150827 18321 ts_manager.cc:194] Registered new tserver with Master: 2bf9c3df528b4758a96f459e463d4e85 (127.5.213.195:44477)
I20260618 22:41:33.151825 18321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:57539
I20260618 22:41:33.152776 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:41:33.154927 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 18401
Jun 18 22:41:33 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:41:33.170379 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:33.155856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39472 (local address 127.5.213.193:39597)
0618 22:41:33.156534 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:33.156540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:33.156563 (+ 23us) server_negotiation.cc:407] Connection header received
0618 22:41:33.156614 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:33.156620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:33.156794 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:33.156919 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:33.158247 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:33.159354 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:33.160791 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:33.161309 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:33.161445 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:33.165388 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:33.165399 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:33.165413 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:33.165445 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:33.168121 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:33.168711 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:33.168717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:33.168719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:33.168858 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:33.169176 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:33.169179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:33.169181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:33.169484 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:33.169705 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:33.169919 (+ 214us) server_negotiation.cc:299] Negotiation successful
0618 22:41:33.170109 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":365,"threads_started":1}
I20260618 22:41:33.171620 18529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:41:33.251638 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 18401
I20260618 22:41:33.264583 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 18540
Jun 18 22:41:33 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.194@KRBTEST.COM
I20260618 22:41:33.280454 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:33.265465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34056 (local address 127.5.213.194:46375)
0618 22:41:33.265946 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:33.265954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:33.265978 (+ 24us) server_negotiation.cc:407] Connection header received
0618 22:41:33.266038 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:33.266044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:33.266269 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:33.266392 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:33.267567 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:33.268615 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:33.269854 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:33.270387 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:33.270478 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:33.274965 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:33.274977 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:33.274992 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:33.275030 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:33.277791 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:33.278427 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:33.278434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:33.278437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:33.278569 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:33.279009 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:33.279012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:33.279014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:33.279369 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:33.279592 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:33.279892 (+ 300us) server_negotiation.cc:299] Negotiation successful
0618 22:41:33.280115 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":239,"threads_started":1}
I20260618 22:41:33.281668 18672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:41:33.363327 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 18540
I20260618 22:41:33.375556 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 18683
Jun 18 22:41:33 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.195@KRBTEST.COM
I20260618 22:41:33.390975 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:33.376428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.5.213.195:44477)
0618 22:41:33.377030 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:33.377037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:33.377066 (+ 29us) server_negotiation.cc:407] Connection header received
0618 22:41:33.377122 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:33.377128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:33.377347 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:33.377474 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:33.378810 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:33.379868 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:33.381170 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:33.381716 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:33.381819 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:33.385949 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:33.385960 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:33.385976 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:33.386006 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:33.388544 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:33.389167 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:33.389173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:33.389176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:33.389299 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:33.389577 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:33.389580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:33.389582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:33.389911 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:33.390151 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:33.390395 (+ 244us) server_negotiation.cc:299] Negotiation successful
0618 22:41:33.390580 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":290,"threads_started":1}
I20260618 22:41:33.392243 18806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:41:33.478966 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 18683
I20260618 22:41:33.490677 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 18289
I20260618 22:41:33.491868 18351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:41:33.597157 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 18289
I20260618 22:41:33.617240 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:39477
--webserver_interface=127.5.213.254
--webserver_port=35399
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:39477
--ranger_config_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.213.212:57679/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.213.212:57679/kms with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:41:33.924160 18822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:33.924451 18822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:33.924546 18822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:33.931648 18822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:41:33.931759 18822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:33.931802 18822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:33.931857 18822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:41:33.931907 18822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:41:33.941767 18822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.213.212:57679/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:39477
--ranger_config_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:39477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=35399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:33.942991 18822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:33.944269 18822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:33.955349 18827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:33.955390 18828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:33.955421 18830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:33.956118 18822 server_base.cc:1034] running on GCE node
I20260618 22:41:33.956858 18822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:33.958144 18822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:33.959334 18822 hybrid_clock.cc:648] HybridClock initialized: now 1781822493959286 us; error 40 us; skew 500 ppm
I20260618 22:41:33.959640 18822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:33 dist-test-slave-lnj3 krb5kdc[17845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822493, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:33.962399 18822 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:41:33.963796 18822 webserver.cc:460] Webserver started at http://127.5.213.254:35399/ using document root <none> and password file <none>
I20260618 22:41:33.964427 18822 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:33.964542 18822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 18 22:41:34 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822493, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
I20260618 22:41:34.049057 18822 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260618 22:41:34.052152 18838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:34.053589 18822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:41:34.053761 18822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "27e98d09d96f45278d9368b95f54004a"
format_stamp: "Formatted at 2026-06-18 22:41:30 on dist-test-slave-lnj3"
server_key: "6a3f5f1236af4013171032a03a85fc16"
server_key_iv: "09d76cee703b6700ebd68b06e1d9d4f5"
server_key_version: "kuduclusterkey@0"
I20260618 22:41:34.054762 18822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:34.069561 18822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:34.074157 18822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:34.074396 18822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:34.091209 18822 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:39477
I20260618 22:41:34.091217 18889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:39477 every 8 connection(s)
I20260618 22:41:34.092624 18822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:41:34.097287 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 18822
I20260618 22:41:34.098295 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.213.212:57679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:39597
--local_ip_for_outbound_sockets=127.5.213.193
--tserver_master_addrs=127.5.213.254:39477
--webserver_port=36091
--webserver_interface=127.5.213.193
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.213.212:57679/kms with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:41:34.100147 18890 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:41:34.104627 18890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a: Bootstrap starting.
I20260618 22:41:34.123150 18890 log.cc:824] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:41:34.131942 18890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:41:34.132413 18890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a: Bootstrap complete.
I20260618 22:41:34.140913 18890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e98d09d96f45278d9368b95f54004a" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 39477 } }
I20260618 22:41:34.141739 18890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27e98d09d96f45278d9368b95f54004a, State: Initialized, Role: FOLLOWER
I20260618 22:41:34.142349 18890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e98d09d96f45278d9368b95f54004a" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 39477 } }
I20260618 22:41:34.142551 18890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:41:34.142686 18890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:41:34.142839 18890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [term 1 FOLLOWER]: Advancing to term 2
I20260618 22:41:34.146502 18890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e98d09d96f45278d9368b95f54004a" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 39477 } }
I20260618 22:41:34.146961 18890 leader_election.cc:302] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27e98d09d96f45278d9368b95f54004a; no voters:
I20260618 22:41:34.147538 18890 leader_election.cc:288] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260618 22:41:34.147686 18894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [term 2 FOLLOWER]: Leader election won for term 2
I20260618 22:41:34.148633 18894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [term 2 LEADER]: Becoming Leader. State: Replica: 27e98d09d96f45278d9368b95f54004a, State: Running, Role: LEADER
I20260618 22:41:34.149621 18890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:41:34.149963 18894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e98d09d96f45278d9368b95f54004a" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 39477 } }
I20260618 22:41:34.152418 18896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27e98d09d96f45278d9368b95f54004a. Latest consensus state: current_term: 2 leader_uuid: "27e98d09d96f45278d9368b95f54004a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e98d09d96f45278d9368b95f54004a" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 39477 } } }
I20260618 22:41:34.152760 18896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [sys.catalog]: This master's current role is: LEADER
I20260618 22:41:34.152559 18895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "27e98d09d96f45278d9368b95f54004a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e98d09d96f45278d9368b95f54004a" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 39477 } } }
I20260618 22:41:34.152897 18895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a [sys.catalog]: This master's current role is: LEADER
I20260618 22:41:34.164716 18903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:41:34.170396 18903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:41:34.172032 18903 catalog_manager.cc:1259] Loaded cluster ID: 48be31faff394372bf3dc32253a22323
I20260618 22:41:34.172197 18903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:41:34.176007 18903 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:41:34.177516 18903 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a: Loaded TSK: 0
I20260618 22:41:34.179136 18903 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:41:34.181674 18890 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260618 22:41:34.538508 18892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:34.539088 18892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:34.539302 18892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:34.551316 18892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:34.551532 18892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:34.551767 18892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:41:34.567446 18892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.213.212:57679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:39597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=36091
--tserver_master_addrs=127.5.213.254:39477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:34.569555 18892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:34.571470 18892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:34.598299 18927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:34.603778 18928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:34.609432 18892 server_base.cc:1034] running on GCE node
W20260618 22:41:34.609766 18930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:34.610594 18892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:34.613478 18892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:34.614720 18892 hybrid_clock.cc:648] HybridClock initialized: now 1781822494614647 us; error 68 us; skew 500 ppm
I20260618 22:41:34.615192 18892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:34 dist-test-slave-lnj3 krb5kdc[17845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822494, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:34.619650 18892 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:41:34.624243 18892 webserver.cc:460] Webserver started at http://127.5.213.193:36091/ using document root <none> and password file <none>
I20260618 22:41:34.625257 18892 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:34.625424 18892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 18 22:41:34 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822494, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
I20260618 22:41:34.659142 18892 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.006s sys 0.004s
I20260618 22:41:34.663281 18938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:34.665429 18892 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260618 22:41:34.665685 18892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "63a51bf4b84f46ec9871b7a150ecca78"
format_stamp: "Formatted at 2026-06-18 22:41:31 on dist-test-slave-lnj3"
server_key: "1a87b944e303a4322f129ac67d07a5f3"
server_key_iv: "87909d5f785bba1e3b260834c2350352"
server_key_version: "kuduclusterkey@0"
I20260618 22:41:34.667188 18892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:34.703071 18892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:34.708438 18892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:34.708817 18892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:34.710130 18892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:41:34.711939 18892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:41:34.712096 18892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260618 22:41:34.712261 18892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:41:34.712386 18892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:34.771442 18892 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:39597
I20260618 22:41:34.771493 19051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:39597 every 8 connection(s)
I20260618 22:41:34.773034 18892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:41:34.783417 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 18892
I20260618 22:41:34.784164 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.213.212:57679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:46375
--local_ip_for_outbound_sockets=127.5.213.194
--tserver_master_addrs=127.5.213.254:39477
--webserver_port=39691
--webserver_interface=127.5.213.194
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.213.212:57679/kms with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:41:34 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822494, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:41:34.800859 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:34.776341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:39477 (local address 127.5.213.193:41565)
0618 22:41:34.777195 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:34.777251 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:34.779350 (+ 2099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:34.781685 (+ 2335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:34.781696 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:34.782342 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:34.783140 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:34.783164 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:34.785502 (+ 2338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:34.785508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:34.787531 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:34.787546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:34.789258 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:34.789264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:34.789530 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:34.790249 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:34.790279 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:34.792378 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:34.796642 (+ 4264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:34.796649 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:34.796663 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:34.797087 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:34.797626 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:34.797632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:34.797636 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:34.797859 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:34.798673 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:34.798682 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:34.799073 (+ 391us) client_negotiation.cc:769] Sending connection context
0618 22:41:34.799390 (+ 317us) client_negotiation.cc:240] Negotiation successful
0618 22:41:34.799840 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":373,"threads_started":1}
I20260618 22:41:34.803432 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:34.776378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:41565 (local address 127.5.213.254:39477)
0618 22:41:34.778000 (+ 1622us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:34.778018 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:34.778066 (+ 48us) server_negotiation.cc:407] Connection header received
0618 22:41:34.780025 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:34.780043 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:34.780542 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:34.780947 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:34.783391 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:34.785303 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:34.787708 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:34.788708 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:34.788887 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:34.792639 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:34.792647 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:34.792663 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:34.792705 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:34.796483 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:34.797249 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:34.797256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:34.797263 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:34.797469 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:34.797983 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:34.797986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:34.797988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:34.798494 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:34.798703 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:34.801699 (+ 2996us) server_negotiation.cc:299] Negotiation successful
0618 22:41:34.802132 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1371,"thread_start_us":1962,"threads_started":1}
I20260618 22:41:34.806952 19052 heartbeater.cc:347] Connected to a master server at 127.5.213.254:39477
I20260618 22:41:34.807289 19052 heartbeater.cc:464] Registering TS with master...
I20260618 22:41:34.808179 19052 heartbeater.cc:511] Master 127.5.213.254:39477 requested a full tablet report, sending...
W20260618 22:41:34.811460 19052 heartbeater.cc:644] Failed to heartbeat to 127.5.213.254:39477 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260618 22:41:34.811712 19052 heartbeater.cc:464] Registering TS with master...
I20260618 22:41:34.811993 19052 heartbeater.cc:511] Master 127.5.213.254:39477 requested a full tablet report, sending...
I20260618 22:41:34.813273 19052 heartbeater.cc:464] Registering TS with master...
I20260618 22:41:34.813557 19052 heartbeater.cc:511] Master 127.5.213.254:39477 requested a full tablet report, sending...
W20260618 22:41:34.814597 19052 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 22:41:35.239543 19056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:35.240015 19056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:35.240200 19056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:35.251089 19056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:35.251243 19056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:35.251482 19056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:41:35.266984 19056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.213.212:57679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:46375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=39691
--tserver_master_addrs=127.5.213.254:39477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:35.268275 19056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:35.269568 19056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:35.286176 19071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:35.287343 19070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:35.292450 19073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:35.292869 19056 server_base.cc:1034] running on GCE node
I20260618 22:41:35.293538 19056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:35.294612 19056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:35.295857 19056 hybrid_clock.cc:648] HybridClock initialized: now 1781822495295807 us; error 42 us; skew 500 ppm
I20260618 22:41:35.296188 19056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:35 dist-test-slave-lnj3 krb5kdc[17845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822495, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:35.299342 19056 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:41:35.301309 19056 webserver.cc:460] Webserver started at http://127.5.213.194:39691/ using document root <none> and password file <none>
I20260618 22:41:35.302310 19056 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:35.302471 19056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 18 22:41:35 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822495, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
I20260618 22:41:35.338151 19056 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.007s sys 0.004s
I20260618 22:41:35.341847 19083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:35.346602 19056 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.003s
I20260618 22:41:35.346912 19056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "f2c8a8d3a3c544e78ae045c76335f58b"
format_stamp: "Formatted at 2026-06-18 22:41:32 on dist-test-slave-lnj3"
server_key: "09329398b6374dd7a37107901038c85b"
server_key_iv: "2b2b132f0c5c7b2fa310e4ec967b9679"
server_key_version: "kuduclusterkey@0"
I20260618 22:41:35.348450 19056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:35.386037 19056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:35.389887 19056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:35.390143 19056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:35.391180 19056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:41:35.392597 19056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:41:35.392688 19056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:35.392813 19056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:41:35.392882 19056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:41:35 dist-test-slave-lnj3 krb5kdc[17845](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781822495, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:35.439661 19056 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:46375
I20260618 22:41:35.439709 19195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:46375 every 8 connection(s)
I20260618 22:41:35.441162 19056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:41:35.447254 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 19056
I20260618 22:41:35.448035 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.213.212:57679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:44477
--local_ip_for_outbound_sockets=127.5.213.195
--tserver_master_addrs=127.5.213.254:39477
--webserver_port=35733
--webserver_interface=127.5.213.195
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.213.212:57679/kms with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:41:35 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822495, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:41:35.499091 19199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:35.456791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:44797 (local address 127.5.213.254:39477)
0618 22:41:35.475527 (+ 18736us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:35.475535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:35.478686 (+ 3151us) server_negotiation.cc:407] Connection header received
0618 22:41:35.480775 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:35.480782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:35.480885 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:35.481003 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:35.483227 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:35.484374 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:35.486783 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:35.487464 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:35.487551 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:35.491958 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:35.491965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:35.491970 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:35.492012 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:35.495544 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:35.496538 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:35.496543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:35.496547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:35.496688 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:35.497170 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:35.497175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:35.497179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:35.497504 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:35.497637 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:35.498476 (+ 839us) server_negotiation.cc:299] Negotiation successful
0618 22:41:35.498722 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18588,"thread_start_us":18468,"threads_started":1}
I20260618 22:41:35.500276 19198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:35.456193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:39477 (local address 127.5.213.194:44797)
0618 22:41:35.477555 (+ 21362us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:35.477627 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:35.480274 (+ 2647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:35.481508 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:35.481522 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:35.482179 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:35.483011 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:35.483036 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:35.484567 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:35.484572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:35.486592 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:35.486620 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:35.487884 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:35.487891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:35.488179 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:35.488953 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:35.488983 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:35.491657 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:35.495712 (+ 4055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:35.495723 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:35.495745 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:35.496409 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:35.496786 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:35.496790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:35.496795 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:35.497057 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:35.497621 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:35.497629 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:35.498061 (+ 432us) client_negotiation.cc:769] Sending connection context
0618 22:41:35.498367 (+ 306us) client_negotiation.cc:240] Negotiation successful
0618 22:41:35.498944 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21027,"thread_start_us":20784,"threads_started":1}
I20260618 22:41:35.501929 19196 heartbeater.cc:347] Connected to a master server at 127.5.213.254:39477
I20260618 22:41:35.502235 19196 heartbeater.cc:464] Registering TS with master...
I20260618 22:41:35.503088 19196 heartbeater.cc:511] Master 127.5.213.254:39477 requested a full tablet report, sending...
I20260618 22:41:35.505156 18855 ts_manager.cc:194] Registered new tserver with Master: f2c8a8d3a3c544e78ae045c76335f58b (127.5.213.194:46375)
I20260618 22:41:35.507515 18855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:44797
I20260618 22:41:35.814778 19052 heartbeater.cc:464] Registering TS with master...
I20260618 22:41:35.815243 19052 heartbeater.cc:511] Master 127.5.213.254:39477 requested a full tablet report, sending...
I20260618 22:41:35.816462 18855 ts_manager.cc:194] Registered new tserver with Master: 63a51bf4b84f46ec9871b7a150ecca78 (127.5.213.193:39597)
I20260618 22:41:35.817646 18855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:41565
Jun 18 22:41:35 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822495, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
W20260618 22:41:35.930054 19200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:35.930528 19200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:35.930723 19200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:35.942307 19200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:35.942517 19200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:35.942751 19200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:41:35.958350 19200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.213.212:57679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:44477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=35733
--tserver_master_addrs=127.5.213.254:39477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:35.960249 19200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:35.961927 19200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:36.005010 19206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:36.007249 19207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:36.017524 19200 server_base.cc:1034] running on GCE node
W20260618 22:41:36.022939 19209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:36.023824 19200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:36.031827 19200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:36.033185 19200 hybrid_clock.cc:648] HybridClock initialized: now 1781822496033080 us; error 102 us; skew 500 ppm
I20260618 22:41:36.033638 19200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:36 dist-test-slave-lnj3 krb5kdc[17845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822496, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:36.037864 19200 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:41:36.058367 19200 webserver.cc:460] Webserver started at http://127.5.213.195:35733/ using document root <none> and password file <none>
I20260618 22:41:36.059502 19200 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:36.059739 19200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 18 22:41:36 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822496, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
I20260618 22:41:36.110313 19200 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.005s sys 0.001s
I20260618 22:41:36.114842 19217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:36.116947 19200 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260618 22:41:36.117226 19200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "2bf9c3df528b4758a96f459e463d4e85"
format_stamp: "Formatted at 2026-06-18 22:41:33 on dist-test-slave-lnj3"
server_key: "4256c11411218877b3f4d0789a063f5e"
server_key_iv: "e5c5a4444b54133304b81e32c5b9e5d7"
server_key_version: "kuduclusterkey@0"
I20260618 22:41:36.118762 19200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:36.165786 19200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:36.171104 19200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:36.171453 19200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:36.193243 19200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:41:36.201344 19200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:41:36.201534 19200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260618 22:41:36.201679 19200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:41:36.201777 19200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:41:36 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822495, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for HTTP/127.5.213.212@KRBTEST.COM
I20260618 22:41:36.407723 19200 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:44477
I20260618 22:41:36.407802 19330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:44477 every 8 connection(s)
I20260618 22:41:36.409678 19200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:41:36.414844 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 19200
I20260618 22:41:36.437392 19334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:36.415769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51700 (local address 127.5.213.254:39477)
0618 22:41:36.419075 (+ 3306us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:36.419082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:36.419111 (+ 29us) server_negotiation.cc:407] Connection header received
0618 22:41:36.419173 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:36.419178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:36.419270 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:36.421301 (+ 2031us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:36.421476 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.422610 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.424969 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.425607 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.429160 (+ 3553us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:36.429414 (+ 254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:36.429420 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:36.429425 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:36.429455 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:36.433170 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:36.434111 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:36.434116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:36.434120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:36.434274 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:36.434767 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:36.434772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:36.434775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:36.435143 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:36.436604 (+ 1461us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:36.436757 (+ 153us) server_negotiation.cc:299] Negotiation successful
0618 22:41:36.436993 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3184,"thread_start_us":238,"threads_started":1}
Jun 18 22:41:36 dist-test-slave-lnj3 krb5kdc[17845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822496, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:41:36.459281 19333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:36.413225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:39477 (local address 127.5.213.195:36559)
0618 22:41:36.426960 (+ 13735us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:36.427011 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:36.429392 (+ 2381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:36.430550 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:36.430563 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:36.431185 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:36.432059 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:36.432087 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.433851 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.433856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:36.436057 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:36.436072 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.437381 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.437389 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:36.437695 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:36.438426 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:36.438458 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:36.449128 (+ 10670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:36.453515 (+ 4387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:36.453527 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:36.453546 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:36.454220 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:36.455241 (+ 1021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:36.455246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:36.455249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:36.455471 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:36.456431 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:36.456439 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:36.456913 (+ 474us) client_negotiation.cc:769] Sending connection context
0618 22:41:36.457204 (+ 291us) client_negotiation.cc:240] Negotiation successful
0618 22:41:36.457817 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13415,"thread_start_us":13142,"threads_started":1}
I20260618 22:41:36.459728 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:36.413858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:36559 (local address 127.5.213.254:39477)
0618 22:41:36.416256 (+ 2398us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:36.416263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:36.427130 (+ 10867us) server_negotiation.cc:407] Connection header received
0618 22:41:36.429752 (+ 2622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:36.429758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:36.429850 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:36.429925 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:36.432299 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.433586 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.436216 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.436912 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.437005 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:36.449442 (+ 12437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:36.449451 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:36.449456 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:36.449499 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:36.453276 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:36.454537 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:36.454542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:36.454545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:36.454728 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:36.455621 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:36.455626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:36.455629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:36.456264 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:36.458940 (+ 2676us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:36.459164 (+ 224us) server_negotiation.cc:299] Negotiation successful
0618 22:41:36.459384 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2268,"thread_start_us":4967,"threads_started":1}
I20260618 22:41:36.461205 19331 heartbeater.cc:347] Connected to a master server at 127.5.213.254:39477
I20260618 22:41:36.461530 19331 heartbeater.cc:464] Registering TS with master...
I20260618 22:41:36.462394 19331 heartbeater.cc:511] Master 127.5.213.254:39477 requested a full tablet report, sending...
I20260618 22:41:36.464092 18855 ts_manager.cc:194] Registered new tserver with Master: 2bf9c3df528b4758a96f459e463d4e85 (127.5.213.195:44477)
I20260618 22:41:36.464923 18855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:36559
I20260618 22:41:36.471314 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:41:36.487566 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:36.474593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.5.213.254:39477)
0618 22:41:36.474777 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:36.474784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:36.475213 (+ 429us) server_negotiation.cc:407] Connection header received
0618 22:41:36.475373 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:36.475379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:36.475469 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:36.475555 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:36.477165 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.477863 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.479314 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.479725 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.479831 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:36.482275 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:36.482280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:36.482284 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:36.482317 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:36.484995 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:36.485725 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:36.485730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:36.485732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:36.485849 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:36.486194 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:36.486199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:36.486202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:36.486488 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:36.486748 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:36.487053 (+ 305us) server_negotiation.cc:299] Negotiation successful
0618 22:41:36.487267 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260618 22:41:36.492565 18855 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:41:36.499969 18855 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:41:36.512313 19196 heartbeater.cc:503] Master 127.5.213.254:39477 was elected leader, sending a full tablet report...
I20260618 22:41:36.537026 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:36.514447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39480 (local address 127.5.213.193:39597)
0618 22:41:36.526780 (+ 12333us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:36.526791 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:36.527721 (+ 930us) server_negotiation.cc:407] Connection header received
0618 22:41:36.527835 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:36.527843 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:36.528125 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:36.528335 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:41:36.529095 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.530733 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.533748 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.535277 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.535401 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:36.535642 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:36.536491 (+ 849us) server_negotiation.cc:299] Negotiation successful
0618 22:41:36.536692 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12089,"thread_start_us":13005,"threads_started":1}
I20260618 22:41:36.537063 19353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:36.514108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:39597 (local address 127.0.0.1:39480)
0618 22:41:36.527214 (+ 13106us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:36.527234 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:36.527380 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:36.528341 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:36.528348 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:36.528378 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:41:36.528890 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:36.528907 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.530900 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.530905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:36.533560 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:36.533570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.535862 (+ 2292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.535869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:36.536179 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:36.536272 (+ 93us) client_negotiation.cc:769] Sending connection context
0618 22:41:36.536422 (+ 150us) client_negotiation.cc:240] Negotiation successful
0618 22:41:36.536653 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12978,"mutex_wait_us":142,"thread_start_us":6398,"threads_started":1}
I20260618 22:41:36.537034 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:36.514664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.5.213.194:46375)
0618 22:41:36.528061 (+ 13397us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:36.528070 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:36.528109 (+ 39us) server_negotiation.cc:407] Connection header received
0618 22:41:36.528204 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:36.528211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:36.528469 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:36.528661 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:41:36.529328 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.530806 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.533725 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.535277 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.535401 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:36.535642 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:36.536580 (+ 938us) server_negotiation.cc:299] Negotiation successful
0618 22:41:36.536735 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13226,"thread_start_us":14142,"threads_started":1}
I20260618 22:41:36.537779 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:36.513959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:46375 (local address 127.0.0.1:34058)
0618 22:41:36.521017 (+ 7058us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:36.521095 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:36.521310 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:36.528682 (+ 7372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:36.528687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:36.528717 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:41:36.529137 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:36.529147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.530942 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.530946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:36.533536 (+ 2590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:36.533559 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.535854 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.535861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:36.536185 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:36.536267 (+ 82us) client_negotiation.cc:769] Sending connection context
0618 22:41:36.536422 (+ 155us) client_negotiation.cc:240] Negotiation successful
0618 22:41:36.537505 (+ 1083us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6722,"mutex_wait_us":1213,"thread_start_us":12991,"threads_started":1}
I20260618 22:41:36.541512 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:36.526426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:44477 (local address 127.0.0.1:48306)
0618 22:41:36.533776 (+ 7350us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:36.533797 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:36.533917 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:36.535688 (+ 1771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:36.535691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:36.535712 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:41:36.536064 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:36.536072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.537760 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.537765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:36.539434 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:36.539445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.540773 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.540780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:36.540946 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:36.540979 (+ 33us) client_negotiation.cc:769] Sending connection context
0618 22:41:36.541054 (+ 75us) client_negotiation.cc:240] Negotiation successful
0618 22:41:36.541176 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7218,"thread_start_us":7053,"threads_started":1}
I20260618 22:41:36.541594 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:36.521985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.5.213.195:44477)
0618 22:41:36.534732 (+ 12747us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:36.534741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:36.534777 (+ 36us) server_negotiation.cc:407] Connection header received
0618 22:41:36.534888 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:36.534898 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:36.535180 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:36.535380 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:41:36.536243 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.537635 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.539569 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.540577 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.540686 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:36.540910 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:36.541114 (+ 204us) server_negotiation.cc:299] Negotiation successful
0618 22:41:36.541274 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12301,"thread_start_us":12154,"threads_started":1}
I20260618 22:41:36.541569 19131 tablet_service.cc:1467] Processing CreateTablet for tablet 01ba9b07cb4e4c599dd3cda0d465bd3c (DEFAULT_TABLE table=test-table [id=389cdc52a5c14443a4f06027c9983d71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:41:36.543087 19131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ba9b07cb4e4c599dd3cda0d465bd3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:41:36.545130 18987 tablet_service.cc:1467] Processing CreateTablet for tablet 01ba9b07cb4e4c599dd3cda0d465bd3c (DEFAULT_TABLE table=test-table [id=389cdc52a5c14443a4f06027c9983d71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:41:36.545143 19266 tablet_service.cc:1467] Processing CreateTablet for tablet 01ba9b07cb4e4c599dd3cda0d465bd3c (DEFAULT_TABLE table=test-table [id=389cdc52a5c14443a4f06027c9983d71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:41:36.546290 18987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ba9b07cb4e4c599dd3cda0d465bd3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:41:36.546466 19266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ba9b07cb4e4c599dd3cda0d465bd3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:41:36.558940 19359 tablet_bootstrap.cc:492] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78: Bootstrap starting.
I20260618 22:41:36.559624 19360 tablet_bootstrap.cc:492] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b: Bootstrap starting.
I20260618 22:41:36.561287 19361 tablet_bootstrap.cc:492] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85: Bootstrap starting.
I20260618 22:41:36.563284 19360 tablet_bootstrap.cc:654] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b: Neither blocks nor log segments found. Creating new log.
I20260618 22:41:36.563277 19359 tablet_bootstrap.cc:654] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78: Neither blocks nor log segments found. Creating new log.
I20260618 22:41:36.564046 19361 tablet_bootstrap.cc:654] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85: Neither blocks nor log segments found. Creating new log.
I20260618 22:41:36.564558 19360 log.cc:824] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:41:36.564558 19359 log.cc:824] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78: Log is configured to *not* fsync() on all Append() calls
I20260618 22:41:36.564980 19361 log.cc:824] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85: Log is configured to *not* fsync() on all Append() calls
I20260618 22:41:36.567592 19359 tablet_bootstrap.cc:492] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78: No bootstrap required, opened a new log
I20260618 22:41:36.567773 19360 tablet_bootstrap.cc:492] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b: No bootstrap required, opened a new log
I20260618 22:41:36.567844 19359 ts_tablet_manager.cc:1397] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260618 22:41:36.568035 19360 ts_tablet_manager.cc:1397] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260618 22:41:36.568923 19361 tablet_bootstrap.cc:492] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85: No bootstrap required, opened a new log
I20260618 22:41:36.569162 19361 ts_tablet_manager.cc:1397] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260618 22:41:36.575336 19359 raft_consensus.cc:348] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c8a8d3a3c544e78ae045c76335f58b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46375 } } peers { permanent_uuid: "63a51bf4b84f46ec9871b7a150ecca78" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39597 } } peers { permanent_uuid: "2bf9c3df528b4758a96f459e463d4e85" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44477 } }
I20260618 22:41:36.575706 19359 raft_consensus.cc:374] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:41:36.575815 19359 raft_consensus.cc:729] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a51bf4b84f46ec9871b7a150ecca78, State: Initialized, Role: FOLLOWER
I20260618 22:41:36.576399 19359 consensus_queue.cc:260] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c8a8d3a3c544e78ae045c76335f58b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46375 } } peers { permanent_uuid: "63a51bf4b84f46ec9871b7a150ecca78" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39597 } } peers { permanent_uuid: "2bf9c3df528b4758a96f459e463d4e85" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44477 } }
I20260618 22:41:36.578264 19360 raft_consensus.cc:348] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c8a8d3a3c544e78ae045c76335f58b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46375 } } peers { permanent_uuid: "63a51bf4b84f46ec9871b7a150ecca78" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39597 } } peers { permanent_uuid: "2bf9c3df528b4758a96f459e463d4e85" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44477 } }
I20260618 22:41:36.578722 19360 raft_consensus.cc:374] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:41:36.578823 19360 raft_consensus.cc:729] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c8a8d3a3c544e78ae045c76335f58b, State: Initialized, Role: FOLLOWER
I20260618 22:41:36.579506 19360 consensus_queue.cc:260] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c8a8d3a3c544e78ae045c76335f58b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46375 } } peers { permanent_uuid: "63a51bf4b84f46ec9871b7a150ecca78" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39597 } } peers { permanent_uuid: "2bf9c3df528b4758a96f459e463d4e85" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44477 } }
I20260618 22:41:36.580061 19361 raft_consensus.cc:348] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c8a8d3a3c544e78ae045c76335f58b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46375 } } peers { permanent_uuid: "63a51bf4b84f46ec9871b7a150ecca78" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39597 } } peers { permanent_uuid: "2bf9c3df528b4758a96f459e463d4e85" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44477 } }
I20260618 22:41:36.581386 19360 ts_tablet_manager.cc:1428] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260618 22:41:36.581861 19361 raft_consensus.cc:374] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:41:36.582022 19361 raft_consensus.cc:729] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf9c3df528b4758a96f459e463d4e85, State: Initialized, Role: FOLLOWER
I20260618 22:41:36.582218 19052 heartbeater.cc:503] Master 127.5.213.254:39477 was elected leader, sending a full tablet report...
I20260618 22:41:36.582894 19359 ts_tablet_manager.cc:1428] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260618 22:41:36.582777 19361 consensus_queue.cc:260] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c8a8d3a3c544e78ae045c76335f58b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46375 } } peers { permanent_uuid: "63a51bf4b84f46ec9871b7a150ecca78" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39597 } } peers { permanent_uuid: "2bf9c3df528b4758a96f459e463d4e85" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44477 } }
I20260618 22:41:36.583820 19331 heartbeater.cc:503] Master 127.5.213.254:39477 was elected leader, sending a full tablet report...
I20260618 22:41:36.584424 19361 ts_tablet_manager.cc:1428] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
W20260618 22:41:36.667835 19332 tablet.cc:2369] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:41:36.716019 19197 tablet.cc:2369] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:41:36.777293 19053 tablet.cc:2369] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:41:36.860136 19365 raft_consensus.cc:482] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:41:36.860453 19365 raft_consensus.cc:504] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c8a8d3a3c544e78ae045c76335f58b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46375 } } peers { permanent_uuid: "63a51bf4b84f46ec9871b7a150ecca78" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39597 } } peers { permanent_uuid: "2bf9c3df528b4758a96f459e463d4e85" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44477 } }
I20260618 22:41:36.862345 19365 leader_election.cc:288] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2c8a8d3a3c544e78ae045c76335f58b (127.5.213.194:46375), 2bf9c3df528b4758a96f459e463d4e85 (127.5.213.195:44477)
I20260618 22:41:36.871124 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:36.863227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:36147 (local address 127.5.213.195:44477)
0618 22:41:36.863425 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:36.863432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:36.864956 (+ 1524us) server_negotiation.cc:407] Connection header received
0618 22:41:36.865190 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:36.865195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:36.865278 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:36.865339 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:41:36.866136 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.867212 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.869440 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.870240 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.870391 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:36.870466 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:36.870768 (+ 302us) server_negotiation.cc:299] Negotiation successful
0618 22:41:36.870896 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260618 22:41:36.871292 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:36.862830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:44477 (local address 127.5.213.193:36147)
0618 22:41:36.864862 (+ 2032us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:36.864879 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:36.865047 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:36.865514 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:36.865519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:36.865539 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:41:36.865890 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:36.865900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.867337 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.867341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:36.869282 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:36.869292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.870431 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.870438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:36.870596 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:36.870619 (+ 23us) client_negotiation.cc:769] Sending connection context
0618 22:41:36.870739 (+ 120us) client_negotiation.cc:240] Negotiation successful
0618 22:41:36.870878 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1916,"thread_start_us":1684,"threads_started":1}
I20260618 22:41:36.871845 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:36.862856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:43905 (local address 127.5.213.194:46375)
0618 22:41:36.863083 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:36.863089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:36.864706 (+ 1617us) server_negotiation.cc:407] Connection header received
0618 22:41:36.864918 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:36.864924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:36.865001 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:36.865084 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:41:36.865810 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.866905 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.869442 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.870671 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.870849 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:36.870925 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:36.871461 (+ 536us) server_negotiation.cc:299] Negotiation successful
0618 22:41:36.871598 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260618 22:41:36.872447 19368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:36.862533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:46375 (local address 127.5.213.193:43905)
0618 22:41:36.864564 (+ 2031us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:36.864599 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:36.864770 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:36.865236 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:36.865240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:36.865273 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:41:36.865652 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:36.865668 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.867062 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.867065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:36.869246 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:36.869262 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.871084 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.871091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:36.871274 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:36.871309 (+ 35us) client_negotiation.cc:769] Sending connection context
0618 22:41:36.872016 (+ 707us) client_negotiation.cc:240] Negotiation successful
0618 22:41:36.872150 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1878,"thread_start_us":1676,"threads_started":1}
I20260618 22:41:36.872581 19286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ba9b07cb4e4c599dd3cda0d465bd3c" candidate_uuid: "63a51bf4b84f46ec9871b7a150ecca78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf9c3df528b4758a96f459e463d4e85" is_pre_election: true
I20260618 22:41:36.873126 19286 raft_consensus.cc:2454] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63a51bf4b84f46ec9871b7a150ecca78 in term 0.
I20260618 22:41:36.873593 19151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ba9b07cb4e4c599dd3cda0d465bd3c" candidate_uuid: "63a51bf4b84f46ec9871b7a150ecca78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2c8a8d3a3c544e78ae045c76335f58b" is_pre_election: true
I20260618 22:41:36.873805 18940 leader_election.cc:302] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf9c3df528b4758a96f459e463d4e85, 63a51bf4b84f46ec9871b7a150ecca78; no voters:
I20260618 22:41:36.874121 19151 raft_consensus.cc:2454] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63a51bf4b84f46ec9871b7a150ecca78 in term 0.
I20260618 22:41:36.874296 19365 raft_consensus.cc:2781] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:41:36.874429 19365 raft_consensus.cc:482] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:41:36.874933 19365 raft_consensus.cc:3037] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:41:36.878016 19365 raft_consensus.cc:504] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c8a8d3a3c544e78ae045c76335f58b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46375 } } peers { permanent_uuid: "63a51bf4b84f46ec9871b7a150ecca78" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39597 } } peers { permanent_uuid: "2bf9c3df528b4758a96f459e463d4e85" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44477 } }
I20260618 22:41:36.878820 19365 leader_election.cc:288] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [CANDIDATE]: Term 1 election: Requested vote from peers f2c8a8d3a3c544e78ae045c76335f58b (127.5.213.194:46375), 2bf9c3df528b4758a96f459e463d4e85 (127.5.213.195:44477)
I20260618 22:41:36.879312 19151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ba9b07cb4e4c599dd3cda0d465bd3c" candidate_uuid: "63a51bf4b84f46ec9871b7a150ecca78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2c8a8d3a3c544e78ae045c76335f58b"
I20260618 22:41:36.879530 19151 raft_consensus.cc:3037] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:41:36.879575 19286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ba9b07cb4e4c599dd3cda0d465bd3c" candidate_uuid: "63a51bf4b84f46ec9871b7a150ecca78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf9c3df528b4758a96f459e463d4e85"
I20260618 22:41:36.879793 19286 raft_consensus.cc:3037] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:41:36.882197 19151 raft_consensus.cc:2454] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63a51bf4b84f46ec9871b7a150ecca78 in term 1.
I20260618 22:41:36.882699 18941 leader_election.cc:302] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63a51bf4b84f46ec9871b7a150ecca78, f2c8a8d3a3c544e78ae045c76335f58b; no voters:
I20260618 22:41:36.882990 19365 raft_consensus.cc:2781] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:41:36.882979 19286 raft_consensus.cc:2454] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63a51bf4b84f46ec9871b7a150ecca78 in term 1.
I20260618 22:41:36.883503 19365 raft_consensus.cc:686] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [term 1 LEADER]: Becoming Leader. State: Replica: 63a51bf4b84f46ec9871b7a150ecca78, State: Running, Role: LEADER
I20260618 22:41:36.884040 19365 consensus_queue.cc:237] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c8a8d3a3c544e78ae045c76335f58b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46375 } } peers { permanent_uuid: "63a51bf4b84f46ec9871b7a150ecca78" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39597 } } peers { permanent_uuid: "2bf9c3df528b4758a96f459e463d4e85" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44477 } }
I20260618 22:41:36.889178 18855 catalog_manager.cc:5495] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63a51bf4b84f46ec9871b7a150ecca78 (127.5.213.193). New cstate: current_term: 1 leader_uuid: "63a51bf4b84f46ec9871b7a150ecca78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c8a8d3a3c544e78ae045c76335f58b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63a51bf4b84f46ec9871b7a150ecca78" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 39597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bf9c3df528b4758a96f459e463d4e85" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44477 } health_report { overall_health: UNKNOWN } } }
I20260618 22:41:36.991828 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:36.986585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39486 (local address 127.5.213.193:39597)
0618 22:41:36.986826 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:36.986831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:36.986850 (+ 19us) server_negotiation.cc:407] Connection header received
0618 22:41:36.987051 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:36.987055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:36.987149 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:36.987282 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:41:36.987656 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.988379 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.989728 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:36.990133 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:36.990237 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:36.990734 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:41:36.991245 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:41:36.991383 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:36.991476 (+ 93us) server_negotiation.cc:299] Negotiation successful
0618 22:41:36.991554 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260618 22:41:36.999986 19286 raft_consensus.cc:1261] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85 [term 1 FOLLOWER]: Refusing update from remote peer 63a51bf4b84f46ec9871b7a150ecca78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:41:36.999986 19151 raft_consensus.cc:1261] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b [term 1 FOLLOWER]: Refusing update from remote peer 63a51bf4b84f46ec9871b7a150ecca78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:41:37.000991 19365 consensus_queue.cc:1035] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf9c3df528b4758a96f459e463d4e85" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:41:37.001303 19370 consensus_queue.cc:1035] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2c8a8d3a3c544e78ae045c76335f58b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:41:37.023559 18855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:41:37.023826 18855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:41:37.027128 18855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 27e98d09d96f45278d9368b95f54004a: Sending DeleteTablet for 3 replicas of tablet 01ba9b07cb4e4c599dd3cda0d465bd3c
I20260618 22:41:37.028198 18987 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ba9b07cb4e4c599dd3cda0d465bd3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:41:37 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:39480
I20260618 22:41:37.028174 19131 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ba9b07cb4e4c599dd3cda0d465bd3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:41:37 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:34058
I20260618 22:41:37.028690 19266 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ba9b07cb4e4c599dd3cda0d465bd3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:41:37 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:48306
I20260618 22:41:37.029026 19384 tablet_replica.cc:330] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78: stopping tablet replica
I20260618 22:41:37.029433 19384 raft_consensus.cc:2229] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:41:37.030054 19384 raft_consensus.cc:2258] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:41:37.030763 19383 tablet_replica.cc:330] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b: stopping tablet replica
I20260618 22:41:37.030915 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 18892
I20260618 22:41:37.031095 19385 tablet_replica.cc:330] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85: stopping tablet replica
I20260618 22:41:37.031155 19383 raft_consensus.cc:2229] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:41:37.031594 19383 raft_consensus.cc:2258] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:41:37.031632 19385 raft_consensus.cc:2229] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:41:37.032325 19385 raft_consensus.cc:2258] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:41:37.033944 19383 ts_tablet_manager.cc:1905] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:41:37.034478 19385 ts_tablet_manager.cc:1905] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:41:37.039547 19384 ts_tablet_manager.cc:1905] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:41:37.042050 19385 ts_tablet_manager.cc:1918] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:41:37.042291 19385 log.cc:1196] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/01ba9b07cb4e4c599dd3cda0d465bd3c
I20260618 22:41:37.043160 19385 ts_tablet_manager.cc:1939] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 2bf9c3df528b4758a96f459e463d4e85: Deleting consensus metadata
I20260618 22:41:37.045799 18840 catalog_manager.cc:4841] TS 2bf9c3df528b4758a96f459e463d4e85 (127.5.213.195:44477): tablet 01ba9b07cb4e4c599dd3cda0d465bd3c (table test-table [id=389cdc52a5c14443a4f06027c9983d71]) successfully deleted
I20260618 22:41:37.046674 19383 ts_tablet_manager.cc:1918] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:41:37.046847 19383 log.cc:1196] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/01ba9b07cb4e4c599dd3cda0d465bd3c
I20260618 22:41:37.047333 19383 ts_tablet_manager.cc:1939] T 01ba9b07cb4e4c599dd3cda0d465bd3c P f2c8a8d3a3c544e78ae045c76335f58b: Deleting consensus metadata
I20260618 22:41:37.049001 19384 ts_tablet_manager.cc:1918] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:41:37.049062 18841 catalog_manager.cc:4841] TS f2c8a8d3a3c544e78ae045c76335f58b (127.5.213.194:46375): tablet 01ba9b07cb4e4c599dd3cda0d465bd3c (table test-table [id=389cdc52a5c14443a4f06027c9983d71]) successfully deleted
I20260618 22:41:37.049167 19384 log.cc:1196] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/01ba9b07cb4e4c599dd3cda0d465bd3c
I20260618 22:41:37.049654 19384 ts_tablet_manager.cc:1939] T 01ba9b07cb4e4c599dd3cda0d465bd3c P 63a51bf4b84f46ec9871b7a150ecca78: Deleting consensus metadata
I20260618 22:41:37.051200 18840 catalog_manager.cc:4841] TS 63a51bf4b84f46ec9871b7a150ecca78 (127.5.213.193:39597): tablet 01ba9b07cb4e4c599dd3cda0d465bd3c (table test-table [id=389cdc52a5c14443a4f06027c9983d71]) successfully deleted
I20260618 22:41:37.061364 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:37.032804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39500 (local address 127.5.213.193:39597)
0618 22:41:37.032957 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:37.032963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:37.042105 (+ 9142us) server_negotiation.cc:407] Connection header received
0618 22:41:37.042317 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:37.042324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:37.042406 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:37.042513 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:37.044223 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:37.045384 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:37.047533 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:37.048206 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:37.048343 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:37.055573 (+ 7230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:37.055583 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:37.055597 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:37.055632 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:37.058374 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:37.059239 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:37.059247 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:37.059251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:37.059419 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:37.059791 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:37.059794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:37.059796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:37.060138 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:37.060343 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:37.060811 (+ 468us) server_negotiation.cc:299] Negotiation successful
0618 22:41:37.060993 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260618 22:41:37.062058 19047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:41:37.175164 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 18892
I20260618 22:41:37.189436 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 19056
I20260618 22:41:37.201646 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:37.190196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34064 (local address 127.5.213.194:46375)
0618 22:41:37.190355 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:37.190359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:37.190376 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:41:37.190433 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:37.190436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:37.190496 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:37.190575 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:37.191909 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:37.192577 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:37.193796 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:37.194197 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:37.194290 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:37.196953 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:37.196965 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:37.196981 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:37.197014 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:37.199543 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:37.200115 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:37.200121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:37.200124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:37.200260 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:37.200527 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:37.200529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:37.200531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:37.200841 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:37.201051 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:37.201241 (+ 190us) server_negotiation.cc:299] Negotiation successful
0618 22:41:37.201388 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:41:37.202242 19191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:41:37.287356 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 19056
I20260618 22:41:37.300410 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 19200
I20260618 22:41:37.312918 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:37.301108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48310 (local address 127.5.213.195:44477)
0618 22:41:37.301274 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:37.301279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:37.301323 (+ 44us) server_negotiation.cc:407] Connection header received
0618 22:41:37.301518 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:37.301522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:37.301582 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:37.301660 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:37.302970 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:37.303659 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:37.304868 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:37.305254 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:37.305357 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:37.307835 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:37.307849 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:37.307864 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:37.307916 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:37.310302 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:37.310993 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:37.311000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:37.311002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:37.311127 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:37.311481 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:37.311484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:37.311487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:37.311853 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:37.312104 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:37.312405 (+ 301us) server_negotiation.cc:299] Negotiation successful
0618 22:41:37.312595 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:41:37.313519 19326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:41:37.399163 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 19200
I20260618 22:41:37.411672 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 18822
I20260618 22:41:37.412863 18885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:41:37.521445 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 18822
I20260618 22:41:37.538900 5975 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260618 22:41:37.873926 5975 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260618 22:41:37.874123 5975 mini_ranger.cc:66] Stopping Ranger...
I20260618 22:41:38.208987 5975 mini_ranger.cc:68] Stopped Ranger
2026-06-18 22:41:38.209 UTC [17875] LOG: received smart shutdown request
2026-06-18 22:41:38.212 UTC [17875] LOG: background worker "logical replication launcher" (PID 17884) exited with exit code 1
2026-06-18 22:41:38.212 UTC [17879] LOG: shutting down
2026-06-18 22:41:38.351 UTC [17875] LOG: database system is shut down
2026-06-18T22:41:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50762 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:41:38 dist-test-slave-lnj3 krb5kdc[19399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:41:38 dist-test-slave-lnj3 krb5kdc[19399](info): set up 2 sockets
Jun 18 22:41:38 dist-test-slave-lnj3 krb5kdc[19399](info): commencing operation
krb5kdc: starting...
W20260618 22:41:39.462736 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:41:39 dist-test-slave-lnj3 krb5kdc[19399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:41:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:41:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:41:39.673985 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:40377
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:40377,127.5.213.253:41315,127.5.213.252:38705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:41:39.989071 19412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:39.989347 19412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:39.989480 19412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:39.996690 19412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:41:39.996798 19412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:39.996871 19412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:39.996946 19412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:41:39.997006 19412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:41:40.006781 19412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:40377,127.5.213.253:41315,127.5.213.252:38705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:40377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:40.008054 19412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:40.009337 19412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:40.020553 19420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:40.020668 19418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:40.020644 19417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:40.021294 19412 server_base.cc:1034] running on GCE node
I20260618 22:41:40.022061 19412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:40.023444 19412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:40.024629 19412 hybrid_clock.cc:648] HybridClock initialized: now 1781822500024585 us; error 34 us; skew 500 ppm
I20260618 22:41:40.024956 19412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:40 dist-test-slave-lnj3 krb5kdc[19399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822500, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:40.027953 19412 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:41:40.029321 19412 webserver.cc:460] Webserver started at http://127.5.213.254:36549/ using document root <none> and password file <none>
I20260618 22:41:40.029970 19412 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:40.030068 19412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:41:40.030324 19412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:41:40.033296 19412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "ad6247c9bf0d435cb24568689b73794b"
format_stamp: "Formatted at 2026-06-18 22:41:40 on dist-test-slave-lnj3"
server_key: "c52a3afeb1ffe935f966a9d85573abed"
server_key_iv: "5e60273923e8659d23f525d1ef18d190"
server_key_version: "encryptionkey@0"
I20260618 22:41:40.034046 19412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "ad6247c9bf0d435cb24568689b73794b"
format_stamp: "Formatted at 2026-06-18 22:41:40 on dist-test-slave-lnj3"
server_key: "c52a3afeb1ffe935f966a9d85573abed"
server_key_iv: "5e60273923e8659d23f525d1ef18d190"
server_key_version: "encryptionkey@0"
I20260618 22:41:40.039067 19412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260618 22:41:40.042327 19427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:40.043605 19412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:41:40.043743 19412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "ad6247c9bf0d435cb24568689b73794b"
format_stamp: "Formatted at 2026-06-18 22:41:40 on dist-test-slave-lnj3"
server_key: "c52a3afeb1ffe935f966a9d85573abed"
server_key_iv: "5e60273923e8659d23f525d1ef18d190"
server_key_version: "encryptionkey@0"
I20260618 22:41:40.043903 19412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:40.060644 19412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:40.064867 19412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:40.065099 19412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:40.081782 19412 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:40377
I20260618 22:41:40.081820 19478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:40377 every 8 connection(s)
I20260618 22:41:40.083205 19412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:41:40.086859 19479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:41:40.092712 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 19412
I20260618 22:41:40.092980 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:41:40.093583 5975 external_mini_cluster.cc:1428] Setting key ef0010d49bd5c31fd34c83f27f5981c7
I20260618 22:41:40.094172 19479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:41:40 dist-test-slave-lnj3 krb5kdc[19399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822500, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.213.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.253@KRBTEST.COM" created.
I20260618 22:41:40.117877 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.097265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:40377 (local address 127.0.0.1:47974)
0618 22:41:40.098091 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.098138 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:40.100018 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:40.101479 (+ 1461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:40.101485 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:40.102077 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:40.102438 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:40.102457 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:40.104607 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:40.104612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:40.105795 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:40.105809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:40.107105 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:40.107111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:40.107250 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:40.107899 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:40.107923 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:40.110380 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:40.113631 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:40.113642 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:40.113652 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:40.114058 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:40.114525 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:40.114529 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:40.114531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:40.114731 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:40.115635 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:40.115642 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:40.115883 (+ 241us) client_negotiation.cc:769] Sending connection context
0618 22:41:40.116167 (+ 284us) client_negotiation.cc:240] Negotiation successful
0618 22:41:40.116820 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":198,"thread_start_us":370,"threads_started":1}
I20260618 22:41:40.117925 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.097265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47974 (local address 127.5.213.254:40377)
0618 22:41:40.100658 (+ 3393us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:40.100666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:40.100704 (+ 38us) server_negotiation.cc:407] Connection header received
0618 22:41:40.100884 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:40.100891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:40.101195 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:40.101413 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:40.102661 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:40.104432 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:40.105946 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:40.106866 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:40.107067 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:40.110682 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:40.110692 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:40.110707 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:40.110737 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:40.113505 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:40.114207 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:40.114217 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:40.114220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:40.114423 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:40.114825 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:40.114829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:40.114831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:40.115508 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:40.115696 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:40.116307 (+ 611us) server_negotiation.cc:299] Negotiation successful
0618 22:41:40.116697 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3228,"spinlock_wait_cycles":19200,"thread_start_us":3051,"threads_started":1}
I20260618 22:41:40.120987 19479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:41:40.122107 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.121708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:41315 (local address 127.0.0.1:38452)
0618 22:41:40.121877 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.121987 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260618 22:41:40.122377 19429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.213.253:41315: connect: Connection refused (error 111)
I20260618 22:41:40.123905 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.123555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:41315 (local address 127.0.0.1:38454)
0618 22:41:40.123691 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.123765 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260618 22:41:40.124111 19479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.253:41315: Network error: Client connection negotiation failed: client connection to 127.5.213.253:41315: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.253.keytab.
Entry for principal kudu/127.5.213.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.253.keytab.
I20260618 22:41:40.141400 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.253:41315
--webserver_interface=127.5.213.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.253.keytab
--principal=kudu/127.5.213.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:40377,127.5.213.253:41315,127.5.213.252:38705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:41:40.173465 19479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } attempt: 1
I20260618 22:41:40.174734 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.174176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:41315 (local address 127.0.0.1:38456)
0618 22:41:40.174396 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.174539 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260618 22:41:40.175753 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.175363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:41315 (local address 127.0.0.1:38460)
0618 22:41:40.175499 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.175585 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260618 22:41:40.175935 19479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.253:41315: Network error: Client connection negotiation failed: client connection to 127.5.213.253:41315: connect: Connection refused (error 111)
I20260618 22:41:40.244194 19479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } attempt: 2
I20260618 22:41:40.245424 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.244882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:41315 (local address 127.0.0.1:38470)
0618 22:41:40.245138 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.245254 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260618 22:41:40.246438 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.246052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:41315 (local address 127.0.0.1:38478)
0618 22:41:40.246201 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.246284 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260618 22:41:40.246654 19479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.253:41315: Network error: Client connection negotiation failed: client connection to 127.5.213.253:41315: connect: Connection refused (error 111)
I20260618 22:41:40.337946 19479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } attempt: 3
I20260618 22:41:40.339221 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.338685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:41315 (local address 127.0.0.1:38490)
0618 22:41:40.338901 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.339035 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260618 22:41:40.340253 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.339844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:41315 (local address 127.0.0.1:38498)
0618 22:41:40.339991 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.340076 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260618 22:41:40.340431 19479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.253:41315: Network error: Client connection negotiation failed: client connection to 127.5.213.253:41315: connect: Connection refused (error 111)
W20260618 22:41:40.451416 19486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:40.451721 19486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:40.451808 19486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:40.458950 19486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:41:40.459048 19486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:40.459098 19486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:40.459156 19486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:41:40.459198 19486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:41:40.468806 19486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:40377,127.5.213.253:41315,127.5.213.252:38705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.253.keytab
--principal=kudu/127.5.213.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.253:41315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.213.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:40.470052 19486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:40.471318 19486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:40.482867 19491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:40.483043 19494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:40.482836 19492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:40.483311 19486 server_base.cc:1034] running on GCE node
I20260618 22:41:40.483641 19479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } attempt: 4
I20260618 22:41:40.484437 19486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:40.484758 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.484243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:41315 (local address 127.0.0.1:38508)
0618 22:41:40.484512 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.484607 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260618 22:41:40.485682 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.485338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:41315 (local address 127.0.0.1:38524)
0618 22:41:40.485480 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.485568 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260618 22:41:40.485882 19479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.253:41315: Network error: Client connection negotiation failed: client connection to 127.5.213.253:41315: connect: Connection refused (error 111)
I20260618 22:41:40.486011 19486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:40.487211 19486 hybrid_clock.cc:648] HybridClock initialized: now 1781822500487156 us; error 46 us; skew 500 ppm
I20260618 22:41:40.487510 19486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:40 dist-test-slave-lnj3 krb5kdc[19399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822500, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:40.490391 19486 init.cc:377] Logged in from keytab as kudu/127.5.213.253@KRBTEST.COM (short username kudu)
I20260618 22:41:40.491799 19486 webserver.cc:460] Webserver started at http://127.5.213.253:45555/ using document root <none> and password file <none>
I20260618 22:41:40.492451 19486 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:40.492563 19486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:41:40.492833 19486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:41:40.495851 19486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-1/data/instance:
uuid: "2a0e452b07464cfdab0274fe3050155b"
format_stamp: "Formatted at 2026-06-18 22:41:40 on dist-test-slave-lnj3"
server_key: "db52ca8ec3129589de362c672286ddf7"
server_key_iv: "34b21a251a44e092650e8d8507e6628f"
server_key_version: "encryptionkey@0"
I20260618 22:41:40.496618 19486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-1/wal/instance:
uuid: "2a0e452b07464cfdab0274fe3050155b"
format_stamp: "Formatted at 2026-06-18 22:41:40 on dist-test-slave-lnj3"
server_key: "db52ca8ec3129589de362c672286ddf7"
server_key_iv: "34b21a251a44e092650e8d8507e6628f"
server_key_version: "encryptionkey@0"
I20260618 22:41:40.501447 19486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:41:40.504808 19501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:40.506084 19486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:41:40.506245 19486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-1/wal
uuid: "2a0e452b07464cfdab0274fe3050155b"
format_stamp: "Formatted at 2026-06-18 22:41:40 on dist-test-slave-lnj3"
server_key: "db52ca8ec3129589de362c672286ddf7"
server_key_iv: "34b21a251a44e092650e8d8507e6628f"
server_key_version: "encryptionkey@0"
I20260618 22:41:40.506417 19486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:40.521323 19486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:40.525517 19486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:40.525784 19486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:40.543924 19486 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.253:41315
I20260618 22:41:40.543942 19552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.253:41315 every 8 connection(s)
I20260618 22:41:40.545368 19486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-1/data/info.pb
I20260618 22:41:40.549954 19553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:41:40.550403 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 19486
I20260618 22:41:40.550566 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-1/wal/instance
I20260618 22:41:40.551036 5975 external_mini_cluster.cc:1428] Setting key f178e0a4e938bfa3f41c064d08acf7dd
I20260618 22:41:40.556454 19553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.213.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.252@KRBTEST.COM" created.
Jun 18 22:41:40 dist-test-slave-lnj3 krb5kdc[19399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822500, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.253@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:41:40.577080 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.559018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.5.213.254:40377)
0618 22:41:40.559171 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:40.559176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:40.560525 (+ 1349us) server_negotiation.cc:407] Connection header received
0618 22:41:40.562386 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:40.562393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:40.562471 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:40.562558 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:40.564078 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:40.564748 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:40.566265 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:40.566704 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:40.566817 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:40.571679 (+ 4862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:40.571684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:40.571687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:40.571722 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:40.574163 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:40.574924 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:40.574930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:40.574933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:40.575048 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:40.575447 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:40.575451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:40.575453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:40.575721 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:40.575853 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:40.576554 (+ 701us) server_negotiation.cc:299] Negotiation successful
0618 22:41:40.576775 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:41:40.578011 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.559614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:40377 (local address 127.0.0.1:47980)
0618 22:41:40.560365 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.560408 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:40.562113 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:40.562826 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:40.562834 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:40.563329 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:40.563905 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:40.563923 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:40.564900 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:40.564904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:40.566112 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:40.566124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:40.566912 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:40.566920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:40.567128 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:40.567737 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:40.567768 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:40.571459 (+ 3691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:40.574300 (+ 2841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:40.574309 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:40.574327 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:40.574816 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:40.575147 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:40.575152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:40.575156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:40.575337 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:40.575849 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:40.575857 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:40.576177 (+ 320us) client_negotiation.cc:769] Sending connection context
0618 22:41:40.576421 (+ 244us) client_negotiation.cc:240] Negotiation successful
0618 22:41:40.576926 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":320,"threads_started":1}
I20260618 22:41:40.579777 19553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:41:40 dist-test-slave-lnj3 krb5kdc[19399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822500, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.253@KRBTEST.COM for kudu/127.5.213.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.252.keytab.
Entry for principal kudu/127.5.213.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.252.keytab.
I20260618 22:41:40.595789 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.580536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:41315 (local address 127.0.0.1:38530)
0618 22:41:40.580716 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.580733 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:40.580874 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:40.581832 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:40.581836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:40.582238 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:40.582366 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:40.582374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:40.583684 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:40.583689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:40.584502 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:40.584512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:40.585306 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:40.585312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:40.585472 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:40.585972 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:40.585989 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:40.590173 (+ 4184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:40.593320 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:40.593326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:40.593330 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:40.593713 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:40.594069 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:40.594074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:40.594078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:40.594217 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:40.594879 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:40.594888 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:40.595020 (+ 132us) client_negotiation.cc:769] Sending connection context
0618 22:41:40.595135 (+ 115us) client_negotiation.cc:240] Negotiation successful
0618 22:41:40.595401 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260618 22:41:40.595878 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.580734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.5.213.253:41315)
0618 22:41:40.581442 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:40.581447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:40.581473 (+ 26us) server_negotiation.cc:407] Connection header received
0618 22:41:40.581515 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:40.581521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:40.581685 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:40.581824 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:40.582503 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:40.583562 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:40.584619 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:40.585152 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:40.585263 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:40.590375 (+ 5112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:40.590388 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:40.590401 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:40.590429 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:40.593208 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:40.593829 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:40.593836 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:40.593838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:40.593969 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:40.594316 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:40.594322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:40.594325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:40.594761 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:40.594952 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:40.595296 (+ 344us) server_negotiation.cc:299] Negotiation successful
0618 22:41:40.595507 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":410,"threads_started":1}
I20260618 22:41:40.595907 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.252:38705
--webserver_interface=127.5.213.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.252.keytab
--principal=kudu/127.5.213.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:40377,127.5.213.253:41315,127.5.213.252:38705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:41:40.598670 19553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:41:40.599795 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.599347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:38705 (local address 127.0.0.1:59920)
0618 22:41:40.599524 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.599640 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260618 22:41:40.600020 19504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
I20260618 22:41:40.601471 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.601161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:38705 (local address 127.0.0.1:59928)
0618 22:41:40.601297 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.601374 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260618 22:41:40.601655 19553 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:38705: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
I20260618 22:41:40.650964 19553 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } attempt: 1
I20260618 22:41:40.652194 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.651651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:38705 (local address 127.0.0.1:59940)
0618 22:41:40.651864 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.651993 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260618 22:41:40.653193 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.652794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:38705 (local address 127.0.0.1:59946)
0618 22:41:40.652961 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.653050 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260618 22:41:40.653389 19553 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:38705: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
I20260618 22:41:40.721621 19553 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } attempt: 2
I20260618 22:41:40.722829 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.722283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:38705 (local address 127.0.0.1:59948)
0618 22:41:40.722499 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.722612 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260618 22:41:40.723834 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.723456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:38705 (local address 127.0.0.1:59962)
0618 22:41:40.723613 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.723698 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260618 22:41:40.724032 19553 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:38705: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
I20260618 22:41:40.785126 19479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } attempt: 5
Jun 18 22:41:40 dist-test-slave-lnj3 krb5kdc[19399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822500, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for kudu/127.5.213.253@KRBTEST.COM
I20260618 22:41:40.797931 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.785937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.5.213.253:41315)
0618 22:41:40.786104 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:40.786110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:40.786135 (+ 25us) server_negotiation.cc:407] Connection header received
0618 22:41:40.786350 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:40.786355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:40.786435 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:40.786520 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:40.787349 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:40.788021 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:40.789182 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:40.789638 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:40.789747 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:40.792885 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:40.792890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:40.792893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:40.792925 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:40.795434 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:40.796169 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:40.796172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:40.796174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:40.796304 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:40.796678 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:40.796680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:40.796682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:40.796991 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:40.797097 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:40.797537 (+ 440us) server_negotiation.cc:299] Negotiation successful
0618 22:41:40.797701 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260618 22:41:40.798120 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.785854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:41315 (local address 127.0.0.1:38538)
0618 22:41:40.786064 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.786085 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:40.786209 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:40.786575 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:40.786581 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:40.787005 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:40.787188 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:40.787198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:40.788151 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:40.788157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:40.789026 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:40.789046 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:40.789787 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:40.789793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:40.789956 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:40.790517 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:40.790534 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:40.792703 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:40.795560 (+ 2857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:40.795565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:40.795569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:40.796061 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:40.796409 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:40.796414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:40.796418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:40.796585 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:40.797113 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:40.797118 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:40.797304 (+ 186us) client_negotiation.cc:769] Sending connection context
0618 22:41:40.797465 (+ 161us) client_negotiation.cc:240] Negotiation successful
0618 22:41:40.797787 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260618 22:41:40.799103 19479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:41:40.800094 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.799744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:38705 (local address 127.0.0.1:59966)
0618 22:41:40.799897 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.799969 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260618 22:41:40.800954 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.800655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:38705 (local address 127.0.0.1:59972)
0618 22:41:40.800798 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.800865 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260618 22:41:40.801144 19479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:38705: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
I20260618 22:41:40.815236 19553 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } attempt: 3
I20260618 22:41:40.816303 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.815791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:38705 (local address 127.0.0.1:59984)
0618 22:41:40.815975 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.816115 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260618 22:41:40.817344 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.816901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:38705 (local address 127.0.0.1:59996)
0618 22:41:40.817134 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.817221 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260618 22:41:40.817636 19553 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:38705: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
I20260618 22:41:40.852334 19479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } attempt: 1
I20260618 22:41:40.853358 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.852908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:38705 (local address 127.0.0.1:59998)
0618 22:41:40.853108 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.853197 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260618 22:41:40.854355 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.854041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:38705 (local address 127.0.0.1:60010)
0618 22:41:40.854166 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.854251 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260618 22:41:40.854552 19479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:38705: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
W20260618 22:41:40.907810 19560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:40.908115 19560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:40.908223 19560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:40.915498 19560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:41:40.915607 19560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:40.915660 19560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:40.915714 19560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:41:40.915756 19560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:41:40.922705 19479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } attempt: 2
I20260618 22:41:40.923791 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.923338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:38705 (local address 127.0.0.1:60020)
0618 22:41:40.923548 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.923634 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260618 22:41:40.924676 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.924382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:38705 (local address 127.0.0.1:60030)
0618 22:41:40.924511 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.924576 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260618 22:41:40.924862 19479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:38705: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
I20260618 22:41:40.925941 19560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:40377,127.5.213.253:41315,127.5.213.252:38705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.252.keytab
--principal=kudu/127.5.213.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.252:38705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.213.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:40.927188 19560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:40.928413 19560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:40.939407 19565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:40.939441 19566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:40.939477 19568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:40.940613 19560 server_base.cc:1034] running on GCE node
I20260618 22:41:40.941346 19560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:40.942710 19560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:40.943907 19560 hybrid_clock.cc:648] HybridClock initialized: now 1781822500943856 us; error 41 us; skew 500 ppm
I20260618 22:41:40.944253 19560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:40 dist-test-slave-lnj3 krb5kdc[19399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822500, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:40.947202 19560 init.cc:377] Logged in from keytab as kudu/127.5.213.252@KRBTEST.COM (short username kudu)
I20260618 22:41:40.948602 19560 webserver.cc:460] Webserver started at http://127.5.213.252:39697/ using document root <none> and password file <none>
I20260618 22:41:40.949258 19560 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:40.949362 19560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:41:40.949638 19560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:41:40.952512 19560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-2/data/instance:
uuid: "440bcda590c64a019e1ea507acd2a045"
format_stamp: "Formatted at 2026-06-18 22:41:40 on dist-test-slave-lnj3"
server_key: "95616f1035352507509ab174ec996bfe"
server_key_iv: "4bb47f2a4b74175573763f970e9d8aab"
server_key_version: "encryptionkey@0"
I20260618 22:41:40.953266 19560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-2/wal/instance:
uuid: "440bcda590c64a019e1ea507acd2a045"
format_stamp: "Formatted at 2026-06-18 22:41:40 on dist-test-slave-lnj3"
server_key: "95616f1035352507509ab174ec996bfe"
server_key_iv: "4bb47f2a4b74175573763f970e9d8aab"
server_key_version: "encryptionkey@0"
I20260618 22:41:40.958050 19560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260618 22:41:40.960902 19553 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } attempt: 4
I20260618 22:41:40.961256 19575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:40.961990 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.961498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:38705 (local address 127.0.0.1:60032)
0618 22:41:40.961701 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.961806 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260618 22:41:40.962904 19560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:41:40.962950 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:40.962560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:38705 (local address 127.0.0.1:60036)
0618 22:41:40.962743 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:40.962824 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260618 22:41:40.963117 19553 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:38705: Network error: Client connection negotiation failed: client connection to 127.5.213.252:38705: connect: Connection refused (error 111)
I20260618 22:41:40.963066 19560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-2/wal
uuid: "440bcda590c64a019e1ea507acd2a045"
format_stamp: "Formatted at 2026-06-18 22:41:40 on dist-test-slave-lnj3"
server_key: "95616f1035352507509ab174ec996bfe"
server_key_iv: "4bb47f2a4b74175573763f970e9d8aab"
server_key_version: "encryptionkey@0"
I20260618 22:41:40.963260 19560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:40.979075 19560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:40.982759 19560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:40.982973 19560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:40.999411 19560 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.252:38705
I20260618 22:41:40.999444 19626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.252:38705 every 8 connection(s)
I20260618 22:41:41.000675 19560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-2/data/info.pb
I20260618 22:41:41.004114 19627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:41:41.005772 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 19560
I20260618 22:41:41.005971 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781822307836422-5975-0/minicluster-data/master-2/wal/instance
I20260618 22:41:41.006536 5975 external_mini_cluster.cc:1428] Setting key bf4b453a1f1f0f2d7ab09b5ec6b341d4
I20260618 22:41:41.013427 19627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:41:41 dist-test-slave-lnj3 krb5kdc[19399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.252@KRBTEST.COM
Jun 18 22:41:41 dist-test-slave-lnj3 krb5kdc[19399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822500, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.252@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:41:41.030272 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:41.008365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60038 (local address 127.5.213.252:38705)
0618 22:41:41.009099 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:41.009114 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:41.009160 (+ 46us) server_negotiation.cc:407] Connection header received
0618 22:41:41.010667 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:41.010695 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:41.011084 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:41.011411 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:41.013120 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.014475 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.016177 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.017177 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.017339 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:41.021444 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:41.021453 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:41.021464 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:41.021502 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:41.025159 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.026481 (+ 1322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.026487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.026494 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.026690 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.027090 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.027094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.027098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.027842 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:41.028119 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:41.028534 (+ 415us) server_negotiation.cc:299] Negotiation successful
0618 22:41:41.028917 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":339,"threads_started":1}
I20260618 22:41:41.030720 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:41.016067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:40377 (local address 127.0.0.1:47996)
0618 22:41:41.016824 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:41.016863 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:41.016995 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:41.017332 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:41.017338 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:41.018096 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:41.018374 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:41.018383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.019383 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.019388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:41.020700 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:41.020712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.021453 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.021459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:41.021628 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:41.022304 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:41.022328 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:41.024879 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:41.028091 (+ 3212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:41.028102 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:41.028109 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:41.028685 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:41.029095 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:41.029099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:41.029101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:41.029218 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:41.029766 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:41.029772 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:41.029922 (+ 150us) client_negotiation.cc:769] Sending connection context
0618 22:41:41.030078 (+ 156us) client_negotiation.cc:240] Negotiation successful
0618 22:41:41.030306 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":377,"threads_started":1}
I20260618 22:41:41.030805 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:41.016097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47996 (local address 127.5.213.254:40377)
0618 22:41:41.016257 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:41.016261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:41.016946 (+ 685us) server_negotiation.cc:407] Connection header received
0618 22:41:41.017110 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:41.017116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:41.017178 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:41.017256 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:41.018524 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.019232 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.020840 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.021255 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.021360 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:41.025087 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:41.025093 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:41.025096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:41.025129 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:41.027975 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.028812 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.028818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.028821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.028979 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.029321 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.029326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.029329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.029635 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:41.029786 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:41.030193 (+ 407us) server_negotiation.cc:299] Negotiation successful
0618 22:41:41.030417 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260618 22:41:41.031037 19479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } attempt: 3
I20260618 22:41:41.033268 19627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:41:41 dist-test-slave-lnj3 krb5kdc[19399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822500, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for kudu/127.5.213.252@KRBTEST.COM
Jun 18 22:41:41 dist-test-slave-lnj3 krb5kdc[19399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822500, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.252@KRBTEST.COM for kudu/127.5.213.253@KRBTEST.COM
Jun 18 22:41:41 dist-test-slave-lnj3 krb5kdc[19399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:41:41.044970 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:41.031716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60040 (local address 127.5.213.252:38705)
0618 22:41:41.031872 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:41.031877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:41.031897 (+ 20us) server_negotiation.cc:407] Connection header received
0618 22:41:41.031944 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:41.031948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:41.032009 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:41.032083 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:41.032949 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.033865 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.035641 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.036241 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.036355 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:41.039890 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:41.039896 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:41.039900 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:41.039940 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:41.042475 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.043260 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.043263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.043265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.043363 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.043705 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.043709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.043711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.043961 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:41.044082 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:41.044476 (+ 394us) server_negotiation.cc:299] Negotiation successful
0618 22:41:41.044650 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260618 22:41:41.044996 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:41.031551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:38705 (local address 127.0.0.1:60040)
0618 22:41:41.031755 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:41.031773 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:41.031867 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:41.032126 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:41.032131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:41.032587 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:41.032795 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:41.032807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.034029 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.034034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:41.035492 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:41.035508 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.036383 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.036389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:41.036555 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:41.037077 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:41.037093 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:41.039679 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:41.042616 (+ 2937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:41.042621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:41.042657 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:41.043138 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:41.043463 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:41.043468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:41.043472 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:41.043601 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:41.044068 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:41.044073 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:41.044221 (+ 148us) client_negotiation.cc:769] Sending connection context
0618 22:41:41.044353 (+ 132us) client_negotiation.cc:240] Negotiation successful
0618 22:41:41.044609 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260618 22:41:41.048092 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:41.033827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:41315 (local address 127.0.0.1:38552)
0618 22:41:41.033976 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:41.033992 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:41.034070 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:41.034589 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:41.034594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:41.034942 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:41.035066 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:41.035073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.036173 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.036179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:41.037514 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:41.037525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.038178 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.038184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:41.038376 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:41.038933 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:41.038955 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:41.041557 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:41.045199 (+ 3642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:41.045204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:41.045207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:41.045716 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:41.046602 (+ 886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:41.046607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:41.046611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:41.046800 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:41.047273 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:41.047278 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:41.047437 (+ 159us) client_negotiation.cc:769] Sending connection context
0618 22:41:41.047543 (+ 106us) client_negotiation.cc:240] Negotiation successful
0618 22:41:41.047784 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260618 22:41:41.048470 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:41.034188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38552 (local address 127.5.213.253:41315)
0618 22:41:41.034351 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:41.034355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:41.034369 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:41:41.034413 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:41.034417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:41.034471 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:41.034549 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:41.035208 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.036045 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.037634 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.038047 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.038148 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:41.041775 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:41.041782 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:41.041787 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:41.041827 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:41.045052 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.046342 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.046347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.046351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.046501 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.046900 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.046905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.046909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.047151 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:41.047286 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:41.047665 (+ 379us) server_negotiation.cc:299] Negotiation successful
0618 22:41:41.047938 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:41:41.049290 19627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:41:41.050165 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:41.034199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47998 (local address 127.5.213.254:40377)
0618 22:41:41.034381 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:41.034386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:41.034402 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:41:41.034450 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:41.034457 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:41.034521 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:41.034611 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:41.036129 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.037283 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.038901 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.039361 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.039465 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:41.044325 (+ 4860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:41.044331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:41.044335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:41.044370 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:41.047452 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.048221 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.048227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.048230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.048413 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.048834 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.048843 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.048846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.049154 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:41.049448 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:41.049687 (+ 239us) server_negotiation.cc:299] Negotiation successful
0618 22:41:41.049877 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260618 22:41:41.051584 19479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b: Bootstrap starting.
I20260618 22:41:41.055071 19479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b: Neither blocks nor log segments found. Creating new log.
I20260618 22:41:41.056259 19479 log.cc:824] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b: Log is configured to *not* fsync() on all Append() calls
Jun 18 22:41:41 dist-test-slave-lnj3 krb5kdc[19399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822500, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.252@KRBTEST.COM for kudu/127.5.213.252@KRBTEST.COM
I20260618 22:41:41.059475 19479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b: No bootstrap required, opened a new log
Jun 18 22:41:41 dist-test-slave-lnj3 krb5kdc[19399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.253@KRBTEST.COM
I20260618 22:41:41.064502 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:41.049936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:38705 (local address 127.0.0.1:60046)
0618 22:41:41.050619 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:41.050666 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:41.050764 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:41.051054 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:41.051058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:41.051471 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:41.051935 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:41.051947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.053286 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.053292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:41.054541 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:41.054554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.055496 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.055502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:41.055701 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:41.056221 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:41.056252 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:41.058675 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:41.062033 (+ 3358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:41.062038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:41.062042 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:41.062469 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:41.062900 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:41.062905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:41.062908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:41.063035 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:41.063628 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:41.063633 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:41.063803 (+ 170us) client_negotiation.cc:769] Sending connection context
0618 22:41:41.063962 (+ 159us) client_negotiation.cc:240] Negotiation successful
0618 22:41:41.064255 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610}
I20260618 22:41:41.065385 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:41.050324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60046 (local address 127.5.213.252:38705)
0618 22:41:41.050471 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:41.050476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:41.050714 (+ 238us) server_negotiation.cc:407] Connection header received
0618 22:41:41.050856 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:41.050861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:41.050936 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:41.051020 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:41.052069 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.053132 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.054688 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.055332 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.055446 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:41.058871 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:41.058876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:41.058880 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:41.058914 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:41.061902 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.062581 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.062586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.062589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.062785 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.063133 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.063137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.063141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.063500 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:41.063650 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:41.064104 (+ 454us) server_negotiation.cc:299] Negotiation successful
0618 22:41:41.065044 (+ 940us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":719,"server-negotiator.queue_time_us":55}
I20260618 22:41:41.066612 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:41.051741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38564 (local address 127.5.213.253:41315)
0618 22:41:41.051901 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:41.051907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:41.051922 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:41:41.051965 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:41.051970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:41.052031 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:41.052111 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:41.053635 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.054331 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.055886 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.056402 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.056504 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:41.060643 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:41.060647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:41.060651 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:41.060679 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:41.063046 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.063657 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.063662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.063665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.063813 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.064262 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.064268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.064272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.064590 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:41.064867 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:41.066030 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0618 22:41:41.066301 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:41:41.069363 19479 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6247c9bf0d435cb24568689b73794b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } } peers { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } } peers { permanent_uuid: "440bcda590c64a019e1ea507acd2a045" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } }
I20260618 22:41:41.069739 19479 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:41:41.069867 19479 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6247c9bf0d435cb24568689b73794b, State: Initialized, Role: FOLLOWER
I20260618 22:41:41.069948 19627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045: Bootstrap starting.
I20260618 22:41:41.070523 19479 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6247c9bf0d435cb24568689b73794b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } } peers { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } } peers { permanent_uuid: "440bcda590c64a019e1ea507acd2a045" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } }
I20260618 22:41:41.071475 19633 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6247c9bf0d435cb24568689b73794b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } } peers { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } } peers { permanent_uuid: "440bcda590c64a019e1ea507acd2a045" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } } }
I20260618 22:41:41.071864 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:41:41.071882 19633 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:41:41.072717 19479 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:41:41.073323 19627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045: Neither blocks nor log segments found. Creating new log.
I20260618 22:41:41.074357 19627 log.cc:824] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045: Log is configured to *not* fsync() on all Append() calls
I20260618 22:41:41.077250 19627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045: No bootstrap required, opened a new log
I20260618 22:41:41.085952 19627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6247c9bf0d435cb24568689b73794b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } } peers { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } } peers { permanent_uuid: "440bcda590c64a019e1ea507acd2a045" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } }
I20260618 22:41:41.086457 19627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:41:41.086596 19627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 440bcda590c64a019e1ea507acd2a045, State: Initialized, Role: FOLLOWER
I20260618 22:41:41.087337 19627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6247c9bf0d435cb24568689b73794b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } } peers { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } } peers { permanent_uuid: "440bcda590c64a019e1ea507acd2a045" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } }
I20260618 22:41:41.097870 19642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6247c9bf0d435cb24568689b73794b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } } peers { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } } peers { permanent_uuid: "440bcda590c64a019e1ea507acd2a045" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } } }
I20260618 22:41:41.098353 19642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:41:41.099232 19627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:41:41.120445 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:41.101184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48000 (local address 127.5.213.254:40377)
0618 22:41:41.101367 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:41.101374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:41.101524 (+ 150us) server_negotiation.cc:407] Connection header received
0618 22:41:41.101814 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:41.101821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:41.101908 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:41.101983 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:41.104598 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.105732 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.108033 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.108781 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.108883 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:41.112698 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:41.112705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:41.112710 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:41.112751 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:41.116507 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.117724 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.117731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.117735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.117932 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.118544 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.118550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.118554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.118931 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:41.119169 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:41.119791 (+ 622us) server_negotiation.cc:299] Negotiation successful
0618 22:41:41.120036 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260618 22:41:41.129371 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:41.103260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.5.213.252:38705)
0618 22:41:41.103418 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:41.103423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:41.111236 (+ 7813us) server_negotiation.cc:407] Connection header received
0618 22:41:41.111495 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:41.111502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:41.111587 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:41.111657 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:41.113763 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.115052 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.117181 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.118115 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.118219 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:41.122013 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:41.122021 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:41.122027 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:41.122074 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:41.125740 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.126527 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.126534 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.126537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.126719 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.127168 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.127172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.127176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.127542 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:41.128665 (+ 1123us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:41.128787 (+ 122us) server_negotiation.cc:299] Negotiation successful
0618 22:41:41.129022 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:41:41.130940 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:41.102095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38576 (local address 127.5.213.253:41315)
0618 22:41:41.102270 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:41.102276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:41.104599 (+ 2323us) server_negotiation.cc:407] Connection header received
0618 22:41:41.105487 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:41.105493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:41.105573 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:41.105637 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:41.110310 (+ 4673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.111393 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.115925 (+ 4532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.116570 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.116659 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:41.123705 (+ 7046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:41.123712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:41.123717 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:41.123774 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:41.126785 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.128497 (+ 1712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.128502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.128505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.128675 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.129193 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.129199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.129203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.129562 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:41.129783 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:41.130260 (+ 477us) server_negotiation.cc:299] Negotiation successful
0618 22:41:41.130488 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260618 22:41:41.137987 19658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:41:41.138198 19658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:41:41.129130 19412 master_runner.cc:186] Error getting master registration for 127.5.213.253:41315: OK, instance_id { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" instance_seqno: 1781822500538722 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 22:41:41.141026 19412 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260618 22:41:41.147768 19665 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:41:41.147883 19665 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:41:41.148664 19560 master_runner.cc:186] Error getting master registration for 127.5.213.253:41315: OK, instance_id { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" instance_seqno: 1781822500538722 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 22:41:41.156366 19560 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260618 22:41:41.234885 19581 tablet.cc:2369] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:41:41.262336 19553 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } attempt: 5
Jun 18 22:41:41 dist-test-slave-lnj3 krb5kdc[19399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822500, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.253@KRBTEST.COM for kudu/127.5.213.252@KRBTEST.COM
I20260618 22:41:41.275172 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:41.262862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:38705 (local address 127.0.0.1:60062)
0618 22:41:41.263098 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:41.263115 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:41.263225 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:41.263539 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:41.263544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:41.263968 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:41.264120 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:41.264129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.265095 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.265101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:41.266062 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:41.266078 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.266890 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.266896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:41.267085 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:41.267668 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:41.267685 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:41.269653 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:41.272607 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:41.272631 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:41.272634 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:41.273023 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:41.273450 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:41.273454 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:41.273456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:41.273616 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:41.274193 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:41.274199 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:41.274330 (+ 131us) client_negotiation.cc:769] Sending connection context
0618 22:41:41.274472 (+ 142us) client_negotiation.cc:240] Negotiation successful
0618 22:41:41.274795 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260618 22:41:41.275261 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:41.262921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.5.213.252:38705)
0618 22:41:41.263101 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:41.263107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:41.263166 (+ 59us) server_negotiation.cc:407] Connection header received
0618 22:41:41.263358 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:41.263362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:41.263418 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:41.263494 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:41.264254 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.264965 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.266193 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:41.266759 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:41.266867 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:41.269812 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:41.269819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:41.269823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:41.269858 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:41.272492 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.273158 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.273163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.273167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.273309 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:41.273754 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:41.273757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:41.273759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:41.274044 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:41.274179 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:41.274576 (+ 397us) server_negotiation.cc:299] Negotiation successful
0618 22:41:41.274889 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260618 22:41:41.280639 19553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b: Bootstrap starting.
I20260618 22:41:41.284020 19553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b: Neither blocks nor log segments found. Creating new log.
I20260618 22:41:41.285187 19553 log.cc:824] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:41:41.288221 19553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b: No bootstrap required, opened a new log
I20260618 22:41:41.295958 19553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6247c9bf0d435cb24568689b73794b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } } peers { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } } peers { permanent_uuid: "440bcda590c64a019e1ea507acd2a045" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } }
I20260618 22:41:41.296365 19553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:41:41.296497 19553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a0e452b07464cfdab0274fe3050155b, State: Initialized, Role: FOLLOWER
I20260618 22:41:41.297114 19553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6247c9bf0d435cb24568689b73794b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } } peers { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } } peers { permanent_uuid: "440bcda590c64a019e1ea507acd2a045" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } }
I20260618 22:41:41.298058 19668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6247c9bf0d435cb24568689b73794b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } } peers { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } } peers { permanent_uuid: "440bcda590c64a019e1ea507acd2a045" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } } }
I20260618 22:41:41.298561 19668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:41:41.299415 19553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b [sys.catalog]: configured and running, proceeding with master startup.
W20260618 22:41:41.318094 19433 tablet.cc:2369] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:41:41.333545 19486 master_runner.cc:263] No leader master found from master 2a0e452b07464cfdab0274fe3050155b
I20260618 22:41:41.333755 19486 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260618 22:41:41.336784 19679 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:41:41.337044 19679 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:41:41.401916 19642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:41:41.402263 19642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6247c9bf0d435cb24568689b73794b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } } peers { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } } peers { permanent_uuid: "440bcda590c64a019e1ea507acd2a045" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } }
I20260618 22:41:41.403937 19642 leader_election.cc:288] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad6247c9bf0d435cb24568689b73794b (127.5.213.254:40377), 2a0e452b07464cfdab0274fe3050155b (127.5.213.253:41315)
I20260618 22:41:41.404954 19454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "440bcda590c64a019e1ea507acd2a045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6247c9bf0d435cb24568689b73794b" is_pre_election: true
I20260618 22:41:41.405616 19454 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 440bcda590c64a019e1ea507acd2a045 in term 0.
I20260618 22:41:41.406885 19528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "440bcda590c64a019e1ea507acd2a045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a0e452b07464cfdab0274fe3050155b" is_pre_election: true
I20260618 22:41:41.407491 19528 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 440bcda590c64a019e1ea507acd2a045 in term 0.
I20260618 22:41:41.408351 19576 leader_election.cc:302] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 440bcda590c64a019e1ea507acd2a045, ad6247c9bf0d435cb24568689b73794b; no voters:
I20260618 22:41:41.409179 19642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:41:41.409346 19642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:41:41.409443 19642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:41:41.412750 19642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6247c9bf0d435cb24568689b73794b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } } peers { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } } peers { permanent_uuid: "440bcda590c64a019e1ea507acd2a045" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } }
I20260618 22:41:41.413982 19454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "440bcda590c64a019e1ea507acd2a045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6247c9bf0d435cb24568689b73794b"
I20260618 22:41:41.414206 19454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:41:41.417670 19454 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 440bcda590c64a019e1ea507acd2a045 in term 1.
I20260618 22:41:41.418195 19576 leader_election.cc:302] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 440bcda590c64a019e1ea507acd2a045, ad6247c9bf0d435cb24568689b73794b; no voters:
I20260618 22:41:41.421461 19528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "440bcda590c64a019e1ea507acd2a045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a0e452b07464cfdab0274fe3050155b"
I20260618 22:41:41.421702 19528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:41:41.427498 19528 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 440bcda590c64a019e1ea507acd2a045 in term 1.
I20260618 22:41:41.430017 19642 leader_election.cc:288] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [CANDIDATE]: Term 1 election: Requested vote from peers ad6247c9bf0d435cb24568689b73794b (127.5.213.254:40377), 2a0e452b07464cfdab0274fe3050155b (127.5.213.253:41315)
I20260618 22:41:41.430445 19642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:41:41.430867 19642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [term 1 LEADER]: Becoming Leader. State: Replica: 440bcda590c64a019e1ea507acd2a045, State: Running, Role: LEADER
I20260618 22:41:41.431654 19642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6247c9bf0d435cb24568689b73794b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } } peers { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } } peers { permanent_uuid: "440bcda590c64a019e1ea507acd2a045" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } }
I20260618 22:41:41.443347 19681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 440bcda590c64a019e1ea507acd2a045. Latest consensus state: current_term: 1 leader_uuid: "440bcda590c64a019e1ea507acd2a045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6247c9bf0d435cb24568689b73794b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } } peers { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } } peers { permanent_uuid: "440bcda590c64a019e1ea507acd2a045" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } } }
I20260618 22:41:41.443881 19681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [sys.catalog]: This master's current role is: LEADER
I20260618 22:41:41.455180 19683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:41:41.457338 19683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:41:41.480170 19528 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b [term 1 FOLLOWER]: Refusing update from remote peer 440bcda590c64a019e1ea507acd2a045: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:41:41.481819 19681 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:41:41.489526 19454 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b [term 1 FOLLOWER]: Refusing update from remote peer 440bcda590c64a019e1ea507acd2a045: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:41:41.490481 19681 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad6247c9bf0d435cb24568689b73794b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:41:41.510142 19668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 440bcda590c64a019e1ea507acd2a045. Latest consensus state: current_term: 1 leader_uuid: "440bcda590c64a019e1ea507acd2a045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6247c9bf0d435cb24568689b73794b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } } peers { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } } peers { permanent_uuid: "440bcda590c64a019e1ea507acd2a045" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } } }
I20260618 22:41:41.510514 19668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:41:41.517308 19684 mvcc.cc:205] Tried to move back new op lower bound from 7298344966049751040 to 7298344965876903936. Current Snapshot: MvccSnapshot[applied={T|T < 7298344966049751040}]
I20260618 22:41:41.518752 19642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "440bcda590c64a019e1ea507acd2a045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6247c9bf0d435cb24568689b73794b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } } peers { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } } peers { permanent_uuid: "440bcda590c64a019e1ea507acd2a045" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } } }
I20260618 22:41:41.519095 19642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [sys.catalog]: This master's current role is: LEADER
I20260618 22:41:41.520596 19633 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 440bcda590c64a019e1ea507acd2a045. Latest consensus state: current_term: 1 leader_uuid: "440bcda590c64a019e1ea507acd2a045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6247c9bf0d435cb24568689b73794b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } } peers { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } } peers { permanent_uuid: "440bcda590c64a019e1ea507acd2a045" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } } }
I20260618 22:41:41.520912 19633 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:41:41.523042 19683 catalog_manager.cc:1347] Generated new cluster ID: 6c5661bfe5d14e2a986502ede9dd6d04
I20260618 22:41:41.523141 19683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:41:41.526208 19681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "440bcda590c64a019e1ea507acd2a045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6247c9bf0d435cb24568689b73794b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } } peers { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } } peers { permanent_uuid: "440bcda590c64a019e1ea507acd2a045" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } } }
I20260618 22:41:41.526873 19681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045 [sys.catalog]: This master's current role is: LEADER
I20260618 22:41:41.526997 19686 mvcc.cc:205] Tried to move back new op lower bound from 7298344966049751040 to 7298344965876903936. Current Snapshot: MvccSnapshot[applied={T|T < 7298344966049751040}]
I20260618 22:41:41.534099 19633 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "440bcda590c64a019e1ea507acd2a045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6247c9bf0d435cb24568689b73794b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } } peers { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } } peers { permanent_uuid: "440bcda590c64a019e1ea507acd2a045" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } } }
I20260618 22:41:41.534502 19633 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:41:41.543030 19668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "440bcda590c64a019e1ea507acd2a045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6247c9bf0d435cb24568689b73794b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40377 } } peers { permanent_uuid: "2a0e452b07464cfdab0274fe3050155b" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 41315 } } peers { permanent_uuid: "440bcda590c64a019e1ea507acd2a045" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 38705 } } }
I20260618 22:41:41.543309 19668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:41:41.599610 19683 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:41:41.604138 19683 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:41:41.632078 19683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 440bcda590c64a019e1ea507acd2a045: Generated new TSK 0
I20260618 22:41:41.633258 19683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:41:42.140106 19658 catalog_manager.cc:1259] Loaded cluster ID: 6c5661bfe5d14e2a986502ede9dd6d04
I20260618 22:41:42.140201 19658 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b: loading cluster ID for follower catalog manager: success
I20260618 22:41:42.142601 19658 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b: acquiring CA information for follower catalog manager: success
I20260618 22:41:42.144366 19658 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ad6247c9bf0d435cb24568689b73794b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 22:41:42.338977 19679 catalog_manager.cc:1259] Loaded cluster ID: 6c5661bfe5d14e2a986502ede9dd6d04
I20260618 22:41:42.339079 19679 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b: loading cluster ID for follower catalog manager: success
I20260618 22:41:42.341434 19679 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b: acquiring CA information for follower catalog manager: success
I20260618 22:41:42.343152 19679 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2a0e452b07464cfdab0274fe3050155b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 22:41:42.748004 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 19412
I20260618 22:41:42.749212 19474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:41:42.855809 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 19412
I20260618 22:41:42.872174 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 19486
I20260618 22:41:42.873211 19548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:41:42.978535 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 19486
I20260618 22:41:42.994484 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 19560
I20260618 22:41:42.995596 19622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:41:43.103279 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:43.102307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:40377 (local address 127.0.0.1:48010)
0618 22:41:43.102924 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:43.103056 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:40377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":403,"threads_started":1}
W20260618 22:41:43.103641 19576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.213.254:40377: connect: Connection refused (error 111)
I20260618 22:41:43.104113 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:43.102687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:41315 (local address 127.0.0.1:38588)
0618 22:41:43.103505 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:43.103591 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":1062,"threads_started":1}
I20260618 22:41:43.104238 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 19560
2026-06-18T22:41:43Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4644 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:41:43 dist-test-slave-lnj3 krb5kdc[19706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:41:43 dist-test-slave-lnj3 krb5kdc[19706](info): set up 2 sockets
Jun 18 22:41:43 dist-test-slave-lnj3 krb5kdc[19706](info): commencing operation
krb5kdc: starting...
W20260618 22:41:44.118908 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:41:44 dist-test-slave-lnj3 krb5kdc[19706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:41:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:41:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:41:44.327616 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:32987
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:32987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:41:44.645037 19719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:44.645313 19719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:44.645440 19719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:44.652441 19719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:41:44.652535 19719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:44.652611 19719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:44.652683 19719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:41:44.652741 19719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:41:44.662186 19719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:32987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:32987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:44.663417 19719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:44.664698 19719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:44.675904 19725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:44.675987 19724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:44.676010 19727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:44.676564 19719 server_base.cc:1034] running on GCE node
I20260618 22:41:44.677289 19719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:44.678604 19719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:44.679873 19719 hybrid_clock.cc:648] HybridClock initialized: now 1781822504679823 us; error 35 us; skew 500 ppm
I20260618 22:41:44.680202 19719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:44 dist-test-slave-lnj3 krb5kdc[19706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822504, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:44.683197 19719 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:41:44.684659 19719 webserver.cc:460] Webserver started at http://127.5.213.254:36443/ using document root <none> and password file <none>
I20260618 22:41:44.685336 19719 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:44.685452 19719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:41:44.685763 19719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:41:44.688714 19719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "4875046ac0a540dda032f2337b8dce61"
format_stamp: "Formatted at 2026-06-18 22:41:44 on dist-test-slave-lnj3"
server_key: "fff8f091004df7f9f522355adaf1f502"
server_key_iv: "38757bb11a5018a751189d1b4a9414ce"
server_key_version: "encryptionkey@0"
I20260618 22:41:44.689437 19719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "4875046ac0a540dda032f2337b8dce61"
format_stamp: "Formatted at 2026-06-18 22:41:44 on dist-test-slave-lnj3"
server_key: "fff8f091004df7f9f522355adaf1f502"
server_key_iv: "38757bb11a5018a751189d1b4a9414ce"
server_key_version: "encryptionkey@0"
I20260618 22:41:44.694368 19719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260618 22:41:44.697669 19734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:44.699043 19719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:41:44.699218 19719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "4875046ac0a540dda032f2337b8dce61"
format_stamp: "Formatted at 2026-06-18 22:41:44 on dist-test-slave-lnj3"
server_key: "fff8f091004df7f9f522355adaf1f502"
server_key_iv: "38757bb11a5018a751189d1b4a9414ce"
server_key_version: "encryptionkey@0"
I20260618 22:41:44.699391 19719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:44.714965 19719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:44.719074 19719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:44.719334 19719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:44.736382 19719 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:32987
I20260618 22:41:44.736394 19785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:32987 every 8 connection(s)
I20260618 22:41:44.737739 19719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:41:44.741325 19786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:41:44.747051 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 19719
I20260618 22:41:44.747262 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:41:44.747922 5975 external_mini_cluster.cc:1428] Setting key d5d2dabb2a67ddd3df081f70f0dbdf28
I20260618 22:41:44.751618 19786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61: Bootstrap starting.
I20260618 22:41:44.755102 19786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61: Neither blocks nor log segments found. Creating new log.
I20260618 22:41:44.756062 19786 log.cc:824] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61: Log is configured to *not* fsync() on all Append() calls
I20260618 22:41:44.758553 19786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61: No bootstrap required, opened a new log
Jun 18 22:41:44 dist-test-slave-lnj3 krb5kdc[19706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:41:44.767586 19786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4875046ac0a540dda032f2337b8dce61" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 32987 } }
I20260618 22:41:44.768030 19786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:41:44.768189 19786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4875046ac0a540dda032f2337b8dce61, State: Initialized, Role: FOLLOWER
I20260618 22:41:44.768847 19786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4875046ac0a540dda032f2337b8dce61" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 32987 } }
I20260618 22:41:44.769134 19786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:41:44.769318 19786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:41:44.769496 19786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:41:44.770927 19789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:44.749825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57062 (local address 127.5.213.254:32987)
0618 22:41:44.750529 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:44.750540 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:44.750578 (+ 38us) server_negotiation.cc:407] Connection header received
0618 22:41:44.752279 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:44.752300 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:44.752670 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:44.752985 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:44.754710 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:44.756047 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:44.757368 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:44.758010 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:44.758178 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:44.762204 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:44.762215 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:44.762238 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:44.762297 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:44.766250 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:44.766993 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:44.767002 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:44.767010 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:44.767221 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:44.767564 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:44.767573 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:44.767576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:44.768317 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:44.768584 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:44.769065 (+ 481us) server_negotiation.cc:299] Negotiation successful
0618 22:41:44.769512 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":316,"threads_started":1}
I20260618 22:41:44.772593 19786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4875046ac0a540dda032f2337b8dce61" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 32987 } }
I20260618 22:41:44.773051 19786 leader_election.cc:302] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4875046ac0a540dda032f2337b8dce61; no voters:
I20260618 22:41:44.773646 19786 leader_election.cc:288] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:41:44.773838 19791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:41:44.774883 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:41:44.775624 19786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:41:44.780618 19791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61 [term 1 LEADER]: Becoming Leader. State: Replica: 4875046ac0a540dda032f2337b8dce61, State: Running, Role: LEADER
I20260618 22:41:44.781175 19791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4875046ac0a540dda032f2337b8dce61" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 32987 } }
I20260618 22:41:44.783676 19792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4875046ac0a540dda032f2337b8dce61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4875046ac0a540dda032f2337b8dce61" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 32987 } } }
I20260618 22:41:44.783771 19793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4875046ac0a540dda032f2337b8dce61. Latest consensus state: current_term: 1 leader_uuid: "4875046ac0a540dda032f2337b8dce61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4875046ac0a540dda032f2337b8dce61" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 32987 } } }
I20260618 22:41:44.784034 19792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61 [sys.catalog]: This master's current role is: LEADER
I20260618 22:41:44.784076 19793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61 [sys.catalog]: This master's current role is: LEADER
W20260618 22:41:44.788681 19805 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:41:44.788791 19805 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:41:44.788911 19806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:41:44.790452 19806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:41:44.797580 19806 catalog_manager.cc:1347] Generated new cluster ID: f6c98e3915624f2da04c229456f7a89d
I20260618 22:41:44.797688 19806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:41:44.810979 19806 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:41:44.812815 19806 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:41:44.820160 19806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4875046ac0a540dda032f2337b8dce61: Generated new TSK 0
I20260618 22:41:44.821195 19806 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:41:44.835371 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 19719
I20260618 22:41:44.836447 19781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:41:44.925915 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 19719
2026-06-18T22:41:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1817 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:41:45 dist-test-slave-lnj3 krb5kdc[19814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:41:45 dist-test-slave-lnj3 krb5kdc[19814](info): set up 2 sockets
Jun 18 22:41:45 dist-test-slave-lnj3 krb5kdc[19814](info): commencing operation
krb5kdc: starting...
W20260618 22:41:45.939064 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:41:46 dist-test-slave-lnj3 krb5kdc[19814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:41:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:41:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:41:46.158924 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:38703
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:38703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:41:46.481302 19827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:46.481621 19827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:46.481766 19827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:46.489048 19827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:41:46.489178 19827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:46.489255 19827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:46.489329 19827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:41:46.489391 19827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:41:46.499253 19827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:38703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:38703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:46.500548 19827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:46.501871 19827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:46.513237 19833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:46.513338 19832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:46.513322 19835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:46.513906 19827 server_base.cc:1034] running on GCE node
I20260618 22:41:46.514727 19827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:46.516134 19827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:46.517351 19827 hybrid_clock.cc:648] HybridClock initialized: now 1781822506517292 us; error 39 us; skew 500 ppm
I20260618 22:41:46.517695 19827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:46 dist-test-slave-lnj3 krb5kdc[19814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822506, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:46.520735 19827 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:41:46.522316 19827 webserver.cc:460] Webserver started at http://127.5.213.254:38019/ using document root <none> and password file <none>
I20260618 22:41:46.523099 19827 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:46.523226 19827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:41:46.523504 19827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:41:46.526484 19827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "fb5c527d935d44cebcdda67d63b35d3b"
format_stamp: "Formatted at 2026-06-18 22:41:46 on dist-test-slave-lnj3"
server_key: "9654ed53e4401b6a6a8b7a47884a8951"
server_key_iv: "fb717e899f44f72d6db74aa783434169"
server_key_version: "encryptionkey@0"
I20260618 22:41:46.527282 19827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "fb5c527d935d44cebcdda67d63b35d3b"
format_stamp: "Formatted at 2026-06-18 22:41:46 on dist-test-slave-lnj3"
server_key: "9654ed53e4401b6a6a8b7a47884a8951"
server_key_iv: "fb717e899f44f72d6db74aa783434169"
server_key_version: "encryptionkey@0"
I20260618 22:41:46.532632 19827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260618 22:41:46.536298 19842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:46.537798 19827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260618 22:41:46.537966 19827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "fb5c527d935d44cebcdda67d63b35d3b"
format_stamp: "Formatted at 2026-06-18 22:41:46 on dist-test-slave-lnj3"
server_key: "9654ed53e4401b6a6a8b7a47884a8951"
server_key_iv: "fb717e899f44f72d6db74aa783434169"
server_key_version: "encryptionkey@0"
I20260618 22:41:46.538164 19827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:46.556262 19827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:46.560786 19827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:46.561074 19827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:46.578389 19827 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:38703
I20260618 22:41:46.578433 19893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:38703 every 8 connection(s)
I20260618 22:41:46.579846 19827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:41:46.583647 19894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:41:46.589373 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 19827
I20260618 22:41:46.589577 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:41:46.590173 5975 external_mini_cluster.cc:1428] Setting key bc7ec779ce6a314040a1506da260a37b
I20260618 22:41:46.594311 19894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b: Bootstrap starting.
I20260618 22:41:46.597990 19894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b: Neither blocks nor log segments found. Creating new log.
I20260618 22:41:46.599133 19894 log.cc:824] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:41:46.602144 19894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b: No bootstrap required, opened a new log
Jun 18 22:41:46 dist-test-slave-lnj3 krb5kdc[19814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:41:46.612408 19894 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5c527d935d44cebcdda67d63b35d3b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 38703 } }
I20260618 22:41:46.612885 19894 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:41:46.613001 19894 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb5c527d935d44cebcdda67d63b35d3b, State: Initialized, Role: FOLLOWER
I20260618 22:41:46.613660 19894 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5c527d935d44cebcdda67d63b35d3b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 38703 } }
I20260618 22:41:46.613931 19894 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:41:46.614068 19894 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:41:46.614244 19894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:41:46.614570 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:46.592194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52114 (local address 127.5.213.254:38703)
0618 22:41:46.592972 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:46.592983 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:46.593022 (+ 39us) server_negotiation.cc:407] Connection header received
0618 22:41:46.594756 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:46.594780 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:46.595179 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:46.595505 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:46.597360 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:46.598780 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:46.600362 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:46.601026 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:46.601164 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:46.605760 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:46.605770 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:46.605795 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:46.605856 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:46.609842 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:46.610568 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:46.610577 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:46.610586 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:46.610873 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:46.611307 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:46.611312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:46.611315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:46.612058 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:46.612353 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:46.612733 (+ 380us) server_negotiation.cc:299] Negotiation successful
0618 22:41:46.613197 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":352,"threads_started":1}
I20260618 22:41:46.616727 19894 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5c527d935d44cebcdda67d63b35d3b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 38703 } }
I20260618 22:41:46.617195 19894 leader_election.cc:302] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb5c527d935d44cebcdda67d63b35d3b; no voters:
I20260618 22:41:46.617727 19894 leader_election.cc:288] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:41:46.617847 19899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:41:46.618453 19899 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 1 LEADER]: Becoming Leader. State: Replica: fb5c527d935d44cebcdda67d63b35d3b, State: Running, Role: LEADER
I20260618 22:41:46.618955 19899 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5c527d935d44cebcdda67d63b35d3b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 38703 } }
I20260618 22:41:46.619565 19894 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:41:46.621461 19900 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb5c527d935d44cebcdda67d63b35d3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5c527d935d44cebcdda67d63b35d3b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 38703 } } }
I20260618 22:41:46.621434 19901 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb5c527d935d44cebcdda67d63b35d3b. Latest consensus state: current_term: 1 leader_uuid: "fb5c527d935d44cebcdda67d63b35d3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5c527d935d44cebcdda67d63b35d3b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 38703 } } }
I20260618 22:41:46.622496 19901 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [sys.catalog]: This master's current role is: LEADER
I20260618 22:41:46.621872 19900 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [sys.catalog]: This master's current role is: LEADER
I20260618 22:41:46.623826 19904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:41:46.628013 19904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:41:46.636257 19904 catalog_manager.cc:1347] Generated new cluster ID: 99462a0954894858b735fe598ffdc5a4
I20260618 22:41:46.636375 19904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:41:46.645705 19904 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:41:46.647492 19904 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:41:46.655987 19904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b: Generated new TSK 0
I20260618 22:41:46.657042 19904 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
I20260618 22:41:46.705634 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:38703
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:41:47.036785 19920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:47.037079 19920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:47.037177 19920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:47.044795 19920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:47.044919 19920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:47.045039 19920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:41:47.054762 19920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:38703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:47.056017 19920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:47.057401 19920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:47.069993 19925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:47.070016 19928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:47.070277 19926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:47.070883 19920 server_base.cc:1034] running on GCE node
I20260618 22:41:47.071506 19920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:47.072454 19920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:47.073661 19920 hybrid_clock.cc:648] HybridClock initialized: now 1781822507073613 us; error 52 us; skew 500 ppm
I20260618 22:41:47.073973 19920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:47 dist-test-slave-lnj3 krb5kdc[19814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822507, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:47.077052 19920 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:41:47.078531 19920 webserver.cc:460] Webserver started at http://127.5.213.193:36379/ using document root <none> and password file <none>
I20260618 22:41:47.079226 19920 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:47.079334 19920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:41:47.079591 19920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:41:47.082509 19920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "efcc780097604e108f1bf64589be096a"
format_stamp: "Formatted at 2026-06-18 22:41:47 on dist-test-slave-lnj3"
server_key: "40e8f261e70aaa6ad33ad8219944774e"
server_key_iv: "c9c7dc8d1321ff0ba7db6fdac4b508ea"
server_key_version: "encryptionkey@0"
I20260618 22:41:47.083307 19920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "efcc780097604e108f1bf64589be096a"
format_stamp: "Formatted at 2026-06-18 22:41:47 on dist-test-slave-lnj3"
server_key: "40e8f261e70aaa6ad33ad8219944774e"
server_key_iv: "c9c7dc8d1321ff0ba7db6fdac4b508ea"
server_key_version: "encryptionkey@0"
I20260618 22:41:47.088451 19920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:41:47.091933 19935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:47.093327 19920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:41:47.093474 19920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "efcc780097604e108f1bf64589be096a"
format_stamp: "Formatted at 2026-06-18 22:41:47 on dist-test-slave-lnj3"
server_key: "40e8f261e70aaa6ad33ad8219944774e"
server_key_iv: "c9c7dc8d1321ff0ba7db6fdac4b508ea"
server_key_version: "encryptionkey@0"
I20260618 22:41:47.093662 19920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:47.132665 19920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:47.137058 19920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:47.137324 19920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:47.138284 19920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:41:47.139696 19920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:41:47.139797 19920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:47.139918 19920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:41:47.139979 19920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:47.170485 19920 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:44055
I20260618 22:41:47.170526 20047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:44055 every 8 connection(s)
I20260618 22:41:47.171777 19920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:41:47.177178 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 19920
I20260618 22:41:47.177387 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:41:47.178128 5975 external_mini_cluster.cc:1428] Setting key 6ac2d84bcd208040f910f20bb36e5d64
Jun 18 22:41:47 dist-test-slave-lnj3 krb5kdc[19814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822507, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:41:47.194237 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:47.174402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:58243 (local address 127.5.213.254:38703)
0618 22:41:47.174885 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:47.174892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:47.175890 (+ 998us) server_negotiation.cc:407] Connection header received
0618 22:41:47.178248 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:47.178255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:47.178336 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:47.178433 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:47.180242 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:47.180964 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:47.183244 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:47.183630 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:47.183748 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:47.187822 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:47.187827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:47.187830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:47.187865 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:47.191108 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:47.191893 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:47.191899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:47.191902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:47.192046 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:47.192470 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:47.192474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:47.192478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:47.192797 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:47.192931 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:47.193647 (+ 716us) server_negotiation.cc:299] Negotiation successful
0618 22:41:47.193871 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":255,"threads_started":1}
I20260618 22:41:47.195178 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:47.174996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:58243)
0618 22:41:47.175720 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:47.175759 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:47.177919 (+ 2160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:47.178742 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:47.178754 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:47.179400 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:47.180066 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:47.180094 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:47.181133 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:47.181138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:47.183077 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:47.183092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:47.183858 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:47.183864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:47.184145 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:47.184880 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:47.184908 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:47.187549 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:47.191262 (+ 3713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:47.191269 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:47.191284 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:47.191765 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:47.192178 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:47.192182 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:47.192184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:47.192371 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:47.192907 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:47.192914 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:47.193272 (+ 358us) client_negotiation.cc:769] Sending connection context
0618 22:41:47.193510 (+ 238us) client_negotiation.cc:240] Negotiation successful
0618 22:41:47.193904 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":272,"threads_started":1}
I20260618 22:41:47.196779 20048 heartbeater.cc:347] Connected to a master server at 127.5.213.254:38703
I20260618 22:41:47.197080 20048 heartbeater.cc:464] Registering TS with master...
I20260618 22:41:47.197943 20048 heartbeater.cc:511] Master 127.5.213.254:38703 requested a full tablet report, sending...
I20260618 22:41:47.199990 19859 ts_manager.cc:194] Registered new tserver with Master: efcc780097604e108f1bf64589be096a (127.5.213.193:44055)
WARNING: no policy specified for kudu/127.5.213.194@KRBTEST.COM; defaulting to no policy
I20260618 22:41:47.201452 19859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:58243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
I20260618 22:41:47.229461 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:38703
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:41:47.547155 20054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:47.547477 20054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:47.547622 20054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:47.554997 20054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:47.555157 20054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:47.555326 20054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:41:47.565322 20054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:38703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:47.566614 20054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:47.567938 20054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:47.586681 20062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:47.586792 20060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:47.587255 20059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:47.588124 20054 server_base.cc:1034] running on GCE node
I20260618 22:41:47.588855 20054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:47.589975 20054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:47.591215 20054 hybrid_clock.cc:648] HybridClock initialized: now 1781822507591151 us; error 57 us; skew 500 ppm
I20260618 22:41:47.591557 20054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:47 dist-test-slave-lnj3 krb5kdc[19814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822507, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:47.594965 20054 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:41:47.596524 20054 webserver.cc:460] Webserver started at http://127.5.213.194:37881/ using document root <none> and password file <none>
I20260618 22:41:47.597294 20054 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:47.597421 20054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:41:47.597743 20054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:41:47.600885 20054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "eb1bc427c6104d1aa18c574cfcbeb05b"
format_stamp: "Formatted at 2026-06-18 22:41:47 on dist-test-slave-lnj3"
server_key: "351753726aae08919cd03dc30c7d0ee5"
server_key_iv: "c6e0403ac895f5689d29255b9e6cfc76"
server_key_version: "encryptionkey@0"
I20260618 22:41:47.601665 20054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "eb1bc427c6104d1aa18c574cfcbeb05b"
format_stamp: "Formatted at 2026-06-18 22:41:47 on dist-test-slave-lnj3"
server_key: "351753726aae08919cd03dc30c7d0ee5"
server_key_iv: "c6e0403ac895f5689d29255b9e6cfc76"
server_key_version: "encryptionkey@0"
I20260618 22:41:47.607199 20054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260618 22:41:47.610822 20069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:47.612283 20054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:41:47.612453 20054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "eb1bc427c6104d1aa18c574cfcbeb05b"
format_stamp: "Formatted at 2026-06-18 22:41:47 on dist-test-slave-lnj3"
server_key: "351753726aae08919cd03dc30c7d0ee5"
server_key_iv: "c6e0403ac895f5689d29255b9e6cfc76"
server_key_version: "encryptionkey@0"
I20260618 22:41:47.612672 20054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:47.635308 20054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:47.639662 20054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:47.639955 20054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:47.640933 20054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:41:47.642453 20054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:41:47.642552 20054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:47.642705 20054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:41:47.642779 20054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:47.675392 20054 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:32851
I20260618 22:41:47.675436 20181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:32851 every 8 connection(s)
I20260618 22:41:47.676759 20054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:41:47.680068 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 20054
I20260618 22:41:47.680279 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:41:47.680807 5975 external_mini_cluster.cc:1428] Setting key 1f3d7958408422bbb6fa17e9265724cf
Jun 18 22:41:47 dist-test-slave-lnj3 krb5kdc[19814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822507, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:41:47.696775 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:47.679258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:50475 (local address 127.5.213.254:38703)
0618 22:41:47.679490 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:47.679494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:47.680746 (+ 1252us) server_negotiation.cc:407] Connection header received
0618 22:41:47.682709 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:47.682714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:47.682797 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:47.682900 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:47.684475 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:47.685455 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:47.687062 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:47.687670 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:47.687784 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:47.691301 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:47.691306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:47.691310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:47.691343 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:47.693761 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:47.694450 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:47.694455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:47.694457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:47.694584 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:47.695020 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:47.695023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:47.695025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:47.695309 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:47.695442 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:47.696137 (+ 695us) server_negotiation.cc:299] Negotiation successful
0618 22:41:47.696399 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260618 22:41:47.697786 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:47.679763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.194:50475)
0618 22:41:47.680571 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:47.680612 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:47.682403 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:47.683154 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:47.683162 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:47.683716 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:47.684302 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:47.684321 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:47.685608 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:47.685612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:47.686938 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:47.686950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:47.687897 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:47.687903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:47.688152 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:47.688840 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:47.688864 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:47.691073 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:47.693892 (+ 2819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:47.693902 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:47.693919 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:47.694328 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:47.694720 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:47.694725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:47.694729 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:47.694907 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:47.695429 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:47.695437 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:47.695776 (+ 339us) client_negotiation.cc:769] Sending connection context
0618 22:41:47.696012 (+ 236us) client_negotiation.cc:240] Negotiation successful
0618 22:41:47.696572 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":318,"threads_started":1}
I20260618 22:41:47.699229 20182 heartbeater.cc:347] Connected to a master server at 127.5.213.254:38703
I20260618 22:41:47.699513 20182 heartbeater.cc:464] Registering TS with master...
I20260618 22:41:47.700203 20182 heartbeater.cc:511] Master 127.5.213.254:38703 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.213.195@KRBTEST.COM; defaulting to no policy
I20260618 22:41:47.701620 19859 ts_manager.cc:194] Registered new tserver with Master: eb1bc427c6104d1aa18c574cfcbeb05b (127.5.213.194:32851)
I20260618 22:41:47.702426 19859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:50475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
I20260618 22:41:47.727478 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:38703
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:41:48.055054 20187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:48.055387 20187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:48.055538 20187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:48.063256 20187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:48.063403 20187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:48.063582 20187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:41:48.073992 20187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:38703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:48.075393 20187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:48.076824 20187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:48.090108 20192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:48.090111 20195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:48.090140 20193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:48.091351 20187 server_base.cc:1034] running on GCE node
I20260618 22:41:48.091982 20187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:48.092958 20187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:48.094202 20187 hybrid_clock.cc:648] HybridClock initialized: now 1781822508094152 us; error 37 us; skew 500 ppm
I20260618 22:41:48.094558 20187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:48 dist-test-slave-lnj3 krb5kdc[19814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822508, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:48.097720 20187 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:41:48.099243 20187 webserver.cc:460] Webserver started at http://127.5.213.195:43757/ using document root <none> and password file <none>
I20260618 22:41:48.099908 20187 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:48.100008 20187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:41:48.100250 20187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:41:48.103139 20187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "4bdb44283a794928bd76eddf3443eaa4"
format_stamp: "Formatted at 2026-06-18 22:41:48 on dist-test-slave-lnj3"
server_key: "dce57b15eb94384fc83048c5f1c6278b"
server_key_iv: "d4f2301607490b78c8aa121e25aa8dd8"
server_key_version: "encryptionkey@0"
I20260618 22:41:48.103892 20187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "4bdb44283a794928bd76eddf3443eaa4"
format_stamp: "Formatted at 2026-06-18 22:41:48 on dist-test-slave-lnj3"
server_key: "dce57b15eb94384fc83048c5f1c6278b"
server_key_iv: "d4f2301607490b78c8aa121e25aa8dd8"
server_key_version: "encryptionkey@0"
I20260618 22:41:48.109143 20187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:41:48.112835 20202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:48.114228 20187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260618 22:41:48.114395 20187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "4bdb44283a794928bd76eddf3443eaa4"
format_stamp: "Formatted at 2026-06-18 22:41:48 on dist-test-slave-lnj3"
server_key: "dce57b15eb94384fc83048c5f1c6278b"
server_key_iv: "d4f2301607490b78c8aa121e25aa8dd8"
server_key_version: "encryptionkey@0"
I20260618 22:41:48.114588 20187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:48.134780 20187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:48.139170 20187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:48.139474 20187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:48.140462 20187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:41:48.141980 20187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:41:48.142089 20187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:48.142225 20187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:41:48.142284 20187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:48.175032 20187 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:44899
I20260618 22:41:48.175069 20314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:44899 every 8 connection(s)
I20260618 22:41:48.176400 20187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:41:48.178532 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 20187
I20260618 22:41:48.178719 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:41:48.179198 5975 external_mini_cluster.cc:1428] Setting key f6cf513fc1be1265e21a62efdbec0da1
Jun 18 22:41:48 dist-test-slave-lnj3 krb5kdc[19814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822508, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:41:48.203390 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:48.179152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:35285 (local address 127.5.213.254:38703)
0618 22:41:48.179376 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:48.179382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:48.180592 (+ 1210us) server_negotiation.cc:407] Connection header received
0618 22:41:48.182861 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:48.182866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:48.182978 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:48.183063 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:48.184553 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.185302 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.186896 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.187287 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.187392 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:48.197694 (+ 10302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:48.197699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:48.197703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:48.197740 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:48.200300 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:48.201041 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:48.201045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:48.201047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:48.201186 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:48.201602 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:48.201606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:48.201608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:48.201883 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:48.202011 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:48.202801 (+ 790us) server_negotiation.cc:299] Negotiation successful
0618 22:41:48.203045 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260618 22:41:48.204306 20048 heartbeater.cc:503] Master 127.5.213.254:38703 was elected leader, sending a full tablet report...
I20260618 22:41:48.204491 20317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:48.179662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.195:35285)
0618 22:41:48.180450 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:48.180503 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:48.182574 (+ 2071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:48.183306 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:48.183314 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:48.183832 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:48.184393 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:48.184410 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.185443 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.185447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:48.186765 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:48.186778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.187498 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.187505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:48.187735 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:48.195129 (+ 7394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:48.195169 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:48.197472 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:48.200442 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:48.200452 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:48.200471 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:48.200915 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:48.201290 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:48.201295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:48.201299 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:48.201490 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:48.202004 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:48.202013 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:48.202393 (+ 380us) client_negotiation.cc:769] Sending connection context
0618 22:41:48.202687 (+ 294us) client_negotiation.cc:240] Negotiation successful
0618 22:41:48.203242 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":322,"threads_started":1}
I20260618 22:41:48.206153 20315 heartbeater.cc:347] Connected to a master server at 127.5.213.254:38703
I20260618 22:41:48.206480 20315 heartbeater.cc:464] Registering TS with master...
I20260618 22:41:48.207320 20315 heartbeater.cc:511] Master 127.5.213.254:38703 requested a full tablet report, sending...
I20260618 22:41:48.208647 19858 ts_manager.cc:194] Registered new tserver with Master: 4bdb44283a794928bd76eddf3443eaa4 (127.5.213.195:44899)
I20260618 22:41:48.209512 19858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:35285
I20260618 22:41:48.219650 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:41:48.236590 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:48.224120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.5.213.254:38703)
0618 22:41:48.224317 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:48.224324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:48.224661 (+ 337us) server_negotiation.cc:407] Connection header received
0618 22:41:48.224855 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:48.224859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:48.224923 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:48.225006 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:48.226484 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.227275 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.228584 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.229018 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.229163 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:48.231465 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:48.231470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:48.231474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:48.231507 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:48.233952 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:48.234662 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:48.234668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:48.234672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:48.234819 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:48.235160 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:48.235165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:48.235167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:48.235512 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:48.235740 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:48.236060 (+ 320us) server_negotiation.cc:299] Negotiation successful
0618 22:41:48.236283 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260618 22:41:48.240897 19858 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:41:48.247676 19858 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:41:48.280920 20329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:48.259858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:32851 (local address 127.0.0.1:44820)
0618 22:41:48.271332 (+ 11474us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:48.271409 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:48.271645 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:48.272944 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:48.272953 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:48.272988 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:41:48.273461 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:48.273482 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.275346 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.275351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:48.277748 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:48.277767 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.279839 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.279846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:48.280034 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:48.280089 (+ 55us) client_negotiation.cc:769] Sending connection context
0618 22:41:48.280241 (+ 152us) client_negotiation.cc:240] Negotiation successful
0618 22:41:48.280449 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11128,"mutex_wait_us":43,"thread_start_us":12190,"threads_started":1}
I20260618 22:41:48.281368 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:48.260289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59536 (local address 127.5.213.193:44055)
0618 22:41:48.273722 (+ 13433us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:48.273730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:48.273757 (+ 27us) server_negotiation.cc:407] Connection header received
0618 22:41:48.273837 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:48.273842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:48.274035 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:48.274203 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:41:48.274887 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.276446 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.278614 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.279952 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.280070 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:48.280312 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:48.280687 (+ 375us) server_negotiation.cc:299] Negotiation successful
0618 22:41:48.280859 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13238,"thread_start_us":13155,"threads_started":1}
I20260618 22:41:48.281541 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:48.259980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:44055 (local address 127.0.0.1:59536)
0618 22:41:48.272350 (+ 12370us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:48.272370 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:48.272478 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:48.274204 (+ 1726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:48.274208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:48.274233 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:41:48.274697 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:48.274709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.276590 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.276596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:48.278449 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:48.278463 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.280388 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.280392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:48.280515 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:48.280537 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:41:48.280634 (+ 97us) client_negotiation.cc:240] Negotiation successful
0618 22:41:48.280778 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12237,"mutex_wait_us":48,"thread_start_us":10864,"threads_started":1}
I20260618 22:41:48.281795 20328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:48.260026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44820 (local address 127.5.213.194:32851)
0618 22:41:48.271992 (+ 11966us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:48.272002 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:48.272035 (+ 33us) server_negotiation.cc:407] Connection header received
0618 22:41:48.272152 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:48.272160 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:48.272435 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:48.272644 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:41:48.273673 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.275187 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.277915 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.279374 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.279507 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:48.279712 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:48.281138 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0618 22:41:48.281310 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11795,"thread_start_us":11676,"threads_started":1}
I20260618 22:41:48.282225 20331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:48.262031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:44899 (local address 127.0.0.1:41620)
0618 22:41:48.273524 (+ 11493us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:48.273545 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:48.273658 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:48.274508 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:48.274513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:48.274541 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:41:48.274964 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:48.274975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.276844 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.276850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:48.278727 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:48.278740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.280715 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.280721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:48.280923 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:48.280954 (+ 31us) client_negotiation.cc:769] Sending connection context
0618 22:41:48.281028 (+ 74us) client_negotiation.cc:240] Negotiation successful
0618 22:41:48.281147 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11375,"mutex_wait_us":88,"thread_start_us":11205,"threads_started":1}
I20260618 22:41:48.281873 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:48.262701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41620 (local address 127.5.213.195:44899)
0618 22:41:48.273041 (+ 10340us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:48.273049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:48.273720 (+ 671us) server_negotiation.cc:407] Connection header received
0618 22:41:48.273826 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:48.273832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:48.274071 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:48.274254 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:41:48.275137 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.276708 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.278876 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.280249 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.280354 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:48.280561 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:48.281383 (+ 822us) server_negotiation.cc:299] Negotiation successful
0618 22:41:48.281551 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10159,"thread_start_us":10012,"threads_started":1}
I20260618 22:41:48.285473 19983 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3ae5066cb5402d884c471920323b58 (DEFAULT_TABLE table=test-table [id=032e9cc22b03421fb390f3604bc70732]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:41:48.285570 20117 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3ae5066cb5402d884c471920323b58 (DEFAULT_TABLE table=test-table [id=032e9cc22b03421fb390f3604bc70732]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:41:48.285946 20250 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3ae5066cb5402d884c471920323b58 (DEFAULT_TABLE table=test-table [id=032e9cc22b03421fb390f3604bc70732]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:41:48.286769 19983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3ae5066cb5402d884c471920323b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:41:48.286932 20117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3ae5066cb5402d884c471920323b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:41:48.287281 20250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3ae5066cb5402d884c471920323b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:41:48.298525 20333 tablet_bootstrap.cc:492] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a: Bootstrap starting.
I20260618 22:41:48.301163 20334 tablet_bootstrap.cc:492] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b: Bootstrap starting.
I20260618 22:41:48.303328 20333 tablet_bootstrap.cc:654] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a: Neither blocks nor log segments found. Creating new log.
I20260618 22:41:48.304612 20334 tablet_bootstrap.cc:654] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b: Neither blocks nor log segments found. Creating new log.
I20260618 22:41:48.304726 20333 log.cc:824] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:41:48.305840 20334 log.cc:824] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:41:48.308097 20333 tablet_bootstrap.cc:492] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a: No bootstrap required, opened a new log
I20260618 22:41:48.308250 20335 tablet_bootstrap.cc:492] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4: Bootstrap starting.
I20260618 22:41:48.308378 20333 ts_tablet_manager.cc:1397] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260618 22:41:48.308838 20334 tablet_bootstrap.cc:492] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b: No bootstrap required, opened a new log
I20260618 22:41:48.309092 20334 ts_tablet_manager.cc:1397] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260618 22:41:48.311127 20335 tablet_bootstrap.cc:654] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4: Neither blocks nor log segments found. Creating new log.
I20260618 22:41:48.312664 20335 log.cc:824] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4: Log is configured to *not* fsync() on all Append() calls
I20260618 22:41:48.315219 20335 tablet_bootstrap.cc:492] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4: No bootstrap required, opened a new log
I20260618 22:41:48.315452 20335 ts_tablet_manager.cc:1397] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260618 22:41:48.318764 20333 raft_consensus.cc:348] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:48.319236 20333 raft_consensus.cc:374] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:41:48.319347 20333 raft_consensus.cc:729] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efcc780097604e108f1bf64589be096a, State: Initialized, Role: FOLLOWER
I20260618 22:41:48.319295 20334 raft_consensus.cc:348] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:48.319784 20334 raft_consensus.cc:374] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:41:48.319885 20334 raft_consensus.cc:729] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb1bc427c6104d1aa18c574cfcbeb05b, State: Initialized, Role: FOLLOWER
I20260618 22:41:48.319980 20333 consensus_queue.cc:260] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:48.320436 20334 consensus_queue.cc:260] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:48.321803 20182 heartbeater.cc:503] Master 127.5.213.254:38703 was elected leader, sending a full tablet report...
I20260618 22:41:48.322510 20334 ts_tablet_manager.cc:1428] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260618 22:41:48.324049 20333 ts_tablet_manager.cc:1428] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260618 22:41:48.324206 20335 raft_consensus.cc:348] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:48.324594 20335 raft_consensus.cc:374] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:41:48.324688 20335 raft_consensus.cc:729] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bdb44283a794928bd76eddf3443eaa4, State: Initialized, Role: FOLLOWER
I20260618 22:41:48.325187 20335 consensus_queue.cc:260] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:48.326166 20315 heartbeater.cc:503] Master 127.5.213.254:38703 was elected leader, sending a full tablet report...
I20260618 22:41:48.326675 20335 ts_tablet_manager.cc:1428] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260618 22:41:48.415891 20340 raft_consensus.cc:482] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:41:48.416177 20340 raft_consensus.cc:504] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:48.417809 20340 leader_election.cc:288] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efcc780097604e108f1bf64589be096a (127.5.213.193:44055), 4bdb44283a794928bd76eddf3443eaa4 (127.5.213.195:44899)
W20260618 22:41:48.424014 20049 tablet.cc:2369] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:41:48.426179 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:48.418379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:44659 (local address 127.5.213.195:44899)
0618 22:41:48.418561 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:48.418567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:48.419125 (+ 558us) server_negotiation.cc:407] Connection header received
0618 22:41:48.419312 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:48.419317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:48.419398 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:48.419498 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:41:48.420197 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.421332 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.424172 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.425310 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.425465 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:48.425539 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:48.425804 (+ 265us) server_negotiation.cc:299] Negotiation successful
0618 22:41:48.425931 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260618 22:41:48.426223 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:48.418181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:44899 (local address 127.5.213.194:44659)
0618 22:41:48.419023 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:48.419040 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:48.419153 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:48.419560 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:48.419564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:48.419580 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:41:48.420055 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:48.420064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.421469 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.421474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:48.424003 (+ 2529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:48.424019 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.425501 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.425506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:48.425623 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:48.425648 (+ 25us) client_negotiation.cc:769] Sending connection context
0618 22:41:48.425750 (+ 102us) client_negotiation.cc:240] Negotiation successful
0618 22:41:48.425844 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":41,"thread_start_us":362,"threads_started":1}
I20260618 22:41:48.427222 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:48.418044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:44055 (local address 127.5.213.194:41171)
0618 22:41:48.418964 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:48.418983 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:48.419186 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:48.419527 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:48.419532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:48.419559 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:41:48.420022 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:48.420034 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.421480 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.421483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:48.424011 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:48.424020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.426330 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.426337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:48.426541 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:48.426570 (+ 29us) client_negotiation.cc:769] Sending connection context
0618 22:41:48.426729 (+ 159us) client_negotiation.cc:240] Negotiation successful
0618 22:41:48.426854 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":141,"spinlock_wait_cycles":23424,"thread_start_us":708,"threads_started":1}
I20260618 22:41:48.427644 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:48.418174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:41171 (local address 127.5.213.193:44055)
0618 22:41:48.418401 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:48.418407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:48.419093 (+ 686us) server_negotiation.cc:407] Connection header received
0618 22:41:48.419333 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:48.419338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:48.419406 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:48.419498 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:41:48.420162 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.421332 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.424856 (+ 3524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.425956 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.426106 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:48.426173 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:48.426775 (+ 602us) server_negotiation.cc:299] Negotiation successful
0618 22:41:48.426902 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260618 22:41:48.427575 20270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3ae5066cb5402d884c471920323b58" candidate_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bdb44283a794928bd76eddf3443eaa4" is_pre_election: true
W20260618 22:41:48.427974 20316 tablet.cc:2369] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:41:48.428154 20270 raft_consensus.cc:2454] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1bc427c6104d1aa18c574cfcbeb05b in term 0.
I20260618 22:41:48.428434 20003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3ae5066cb5402d884c471920323b58" candidate_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efcc780097604e108f1bf64589be096a" is_pre_election: true
W20260618 22:41:48.428661 20183 tablet.cc:2369] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:41:48.428912 20071 leader_election.cc:302] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bdb44283a794928bd76eddf3443eaa4, eb1bc427c6104d1aa18c574cfcbeb05b; no voters:
I20260618 22:41:48.428952 20003 raft_consensus.cc:2454] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1bc427c6104d1aa18c574cfcbeb05b in term 0.
I20260618 22:41:48.429275 20340 raft_consensus.cc:2781] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:41:48.429407 20340 raft_consensus.cc:482] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:41:48.429512 20340 raft_consensus.cc:3037] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:41:48.431841 20340 raft_consensus.cc:504] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:48.432454 20340 leader_election.cc:288] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [CANDIDATE]: Term 1 election: Requested vote from peers efcc780097604e108f1bf64589be096a (127.5.213.193:44055), 4bdb44283a794928bd76eddf3443eaa4 (127.5.213.195:44899)
I20260618 22:41:48.432801 20003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3ae5066cb5402d884c471920323b58" candidate_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efcc780097604e108f1bf64589be096a"
I20260618 22:41:48.432955 20270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3ae5066cb5402d884c471920323b58" candidate_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bdb44283a794928bd76eddf3443eaa4"
I20260618 22:41:48.433010 20003 raft_consensus.cc:3037] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:41:48.433156 20270 raft_consensus.cc:3037] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:41:48.436151 20270 raft_consensus.cc:2454] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1bc427c6104d1aa18c574cfcbeb05b in term 1.
I20260618 22:41:48.436153 20003 raft_consensus.cc:2454] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1bc427c6104d1aa18c574cfcbeb05b in term 1.
I20260618 22:41:48.436568 20072 leader_election.cc:302] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eb1bc427c6104d1aa18c574cfcbeb05b, efcc780097604e108f1bf64589be096a; no voters:
I20260618 22:41:48.436836 20340 raft_consensus.cc:2781] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:41:48.437255 20340 raft_consensus.cc:686] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [term 1 LEADER]: Becoming Leader. State: Replica: eb1bc427c6104d1aa18c574cfcbeb05b, State: Running, Role: LEADER
I20260618 22:41:48.437759 20340 consensus_queue.cc:237] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:48.442238 19857 catalog_manager.cc:5495] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b reported cstate change: term changed from 0 to 1, leader changed from <none> to eb1bc427c6104d1aa18c574cfcbeb05b (127.5.213.194). New cstate: current_term: 1 leader_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } health_report { overall_health: UNKNOWN } } }
I20260618 22:41:48.477183 20328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:48.472253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44830 (local address 127.5.213.194:32851)
0618 22:41:48.472432 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:48.472437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:48.472455 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:41:48.472688 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:48.472703 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:48.472783 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:48.472908 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:41:48.473222 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.473962 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.475318 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.475721 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.475812 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:48.476250 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:41:48.476760 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:41:48.476877 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:48.476916 (+ 39us) server_negotiation.cc:299] Negotiation successful
0618 22:41:48.476979 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 22:41:48.485054 20270 raft_consensus.cc:1261] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4 [term 1 FOLLOWER]: Refusing update from remote peer eb1bc427c6104d1aa18c574cfcbeb05b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:41:48.485054 20003 raft_consensus.cc:1261] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Refusing update from remote peer eb1bc427c6104d1aa18c574cfcbeb05b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:41:48.486152 20344 consensus_queue.cc:1035] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:41:48.486485 20340 consensus_queue.cc:1035] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [LEADER]: Connected to new peer: Peer: permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:41:48.497885 20348 mvcc.cc:205] Tried to move back new op lower bound from 7298344994744455168 to 7298344994564751360. Current Snapshot: MvccSnapshot[applied={T|T < 7298344994744455168}]
I20260618 22:41:48.514014 19858 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:41:48.514262 19858 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:41:48.517814 19858 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b: Sending DeleteTablet for 3 replicas of tablet 7d3ae5066cb5402d884c471920323b58
I20260618 22:41:48.518936 20117 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d3ae5066cb5402d884c471920323b58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:41:48 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:44820
I20260618 22:41:48.519188 19983 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d3ae5066cb5402d884c471920323b58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:41:48 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:59536
I20260618 22:41:48.519192 20250 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d3ae5066cb5402d884c471920323b58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:41:48 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:41620
I20260618 22:41:48.520475 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 19920
I20260618 22:41:48.530141 20358 tablet_replica.cc:330] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b: stopping tablet replica
I20260618 22:41:48.530810 20358 raft_consensus.cc:2229] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:41:48.531641 20358 raft_consensus.cc:2258] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:41:48.533994 20358 ts_tablet_manager.cc:1905] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:41:48.541393 20358 ts_tablet_manager.cc:1918] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:41:48.541632 20358 log.cc:1196] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/7d3ae5066cb5402d884c471920323b58
I20260618 22:41:48.542233 20358 ts_tablet_manager.cc:1939] T 7d3ae5066cb5402d884c471920323b58 P eb1bc427c6104d1aa18c574cfcbeb05b: Deleting consensus metadata
I20260618 22:41:48.544139 19843 catalog_manager.cc:4841] TS eb1bc427c6104d1aa18c574cfcbeb05b (127.5.213.194:32851): tablet 7d3ae5066cb5402d884c471920323b58 (table test-table [id=032e9cc22b03421fb390f3604bc70732]) successfully deleted
I20260618 22:41:48.545077 20360 tablet_replica.cc:330] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a: stopping tablet replica
I20260618 22:41:48.545585 20360 raft_consensus.cc:2229] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:41:48.546231 20359 tablet_replica.cc:330] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4: stopping tablet replica
I20260618 22:41:48.546681 20359 raft_consensus.cc:2229] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:41:48.547160 20359 raft_consensus.cc:2258] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:41:48.547192 20360 raft_consensus.cc:2258] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:41:48.549090 20360 ts_tablet_manager.cc:1905] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:41:48.550046 20359 ts_tablet_manager.cc:1905] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:41:48.553612 20360 ts_tablet_manager.cc:1918] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:41:48.553746 20360 log.cc:1196] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/7d3ae5066cb5402d884c471920323b58
I20260618 22:41:48.554256 20360 ts_tablet_manager.cc:1939] T 7d3ae5066cb5402d884c471920323b58 P efcc780097604e108f1bf64589be096a: Deleting consensus metadata
I20260618 22:41:48.555977 19845 catalog_manager.cc:4841] TS efcc780097604e108f1bf64589be096a (127.5.213.193:44055): tablet 7d3ae5066cb5402d884c471920323b58 (table test-table [id=032e9cc22b03421fb390f3604bc70732]) successfully deleted
I20260618 22:41:48.556717 20359 ts_tablet_manager.cc:1918] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:41:48.556953 20359 log.cc:1196] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/7d3ae5066cb5402d884c471920323b58
I20260618 22:41:48.557440 20359 ts_tablet_manager.cc:1939] T 7d3ae5066cb5402d884c471920323b58 P 4bdb44283a794928bd76eddf3443eaa4: Deleting consensus metadata
I20260618 22:41:48.559254 19844 catalog_manager.cc:4841] TS 4bdb44283a794928bd76eddf3443eaa4 (127.5.213.195:44899): tablet 7d3ae5066cb5402d884c471920323b58 (table test-table [id=032e9cc22b03421fb390f3604bc70732]) successfully deleted
Jun 18 22:41:48 dist-test-slave-lnj3 krb5kdc[19814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:41:48.567312 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:48.521713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59552 (local address 127.5.213.193:44055)
0618 22:41:48.521893 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:48.521899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:48.544798 (+ 22899us) server_negotiation.cc:407] Connection header received
0618 22:41:48.545207 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:48.545214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:48.545305 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:48.545385 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:48.547240 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.548397 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.555804 (+ 7407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.556402 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.556476 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:48.561509 (+ 5033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:48.561522 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:48.561536 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:48.561578 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:48.564375 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:48.565211 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:48.565224 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:48.565226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:48.565355 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:48.565720 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:48.565723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:48.565725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:48.566065 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:48.566278 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:48.566738 (+ 460us) server_negotiation.cc:299] Negotiation successful
0618 22:41:48.566953 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:41:48.568019 20043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:41:48.668646 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 19920
I20260618 22:41:48.683338 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 20054
Jun 18 22:41:48 dist-test-slave-lnj3 krb5kdc[19814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.194@KRBTEST.COM
I20260618 22:41:48.698741 20328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:48.684222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44842 (local address 127.5.213.194:32851)
0618 22:41:48.684380 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:48.684386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:48.684410 (+ 24us) server_negotiation.cc:407] Connection header received
0618 22:41:48.684587 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:48.684592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:48.684666 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:48.684768 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:48.685992 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.687258 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.688671 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.689082 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.689193 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:48.693125 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:48.693136 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:48.693153 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:48.693194 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:48.696054 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:48.696727 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:48.696734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:48.696736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:48.696873 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:48.697223 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:48.697227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:48.697229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:48.697580 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:48.697894 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:48.698146 (+ 252us) server_negotiation.cc:299] Negotiation successful
0618 22:41:48.698348 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 22:41:48.699448 20177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:41:48.811731 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 20054
I20260618 22:41:48.826562 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 20187
Jun 18 22:41:48 dist-test-slave-lnj3 krb5kdc[19814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.195@KRBTEST.COM
I20260618 22:41:48.840971 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:48.827375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41630 (local address 127.5.213.195:44899)
0618 22:41:48.827548 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:48.827558 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:48.827615 (+ 57us) server_negotiation.cc:407] Connection header received
0618 22:41:48.827709 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:48.827712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:48.827775 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:48.827856 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:48.829096 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.830011 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.831434 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:48.831892 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:48.831989 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:48.836074 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:48.836089 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:48.836106 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:48.836143 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:48.838545 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:48.839239 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:48.839246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:48.839249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:48.839383 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:48.839751 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:48.839754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:48.839755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:48.840074 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:48.840307 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:48.840555 (+ 248us) server_negotiation.cc:299] Negotiation successful
0618 22:41:48.840721 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:41:48.841698 20310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:41:48.928594 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 20187
I20260618 22:41:48.941740 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 19827
I20260618 22:41:48.942837 19889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:41:49.052014 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 19827
I20260618 22:41:49.069243 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:38703
--webserver_interface=127.5.213.254
--webserver_port=38019
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:38703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:41:49.391323 20366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:49.391631 20366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:49.391768 20366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:49.399101 20366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:41:49.399207 20366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:49.399273 20366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:49.399349 20366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:41:49.399408 20366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:41:49.409368 20366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:38703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:38703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=38019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:49.410709 20366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:49.411971 20366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:49.433039 20373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:49.433022 20372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:49.433143 20375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:49.433812 20366 server_base.cc:1034] running on GCE node
I20260618 22:41:49.434751 20366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:49.436375 20366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:49.437599 20366 hybrid_clock.cc:648] HybridClock initialized: now 1781822509437542 us; error 42 us; skew 500 ppm
I20260618 22:41:49.438010 20366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:49 dist-test-slave-lnj3 krb5kdc[19814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822509, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:49.441447 20366 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:41:49.443029 20366 webserver.cc:460] Webserver started at http://127.5.213.254:38019/ using document root <none> and password file <none>
I20260618 22:41:49.443717 20366 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:49.443835 20366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:41:49.449673 20366 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260618 22:41:49.452492 20382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:49.453850 20366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260618 22:41:49.454003 20366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "fb5c527d935d44cebcdda67d63b35d3b"
format_stamp: "Formatted at 2026-06-18 22:41:46 on dist-test-slave-lnj3"
server_key: "9654ed53e4401b6a6a8b7a47884a8951"
server_key_iv: "fb717e899f44f72d6db74aa783434169"
server_key_version: "encryptionkey@0"
I20260618 22:41:49.455119 20366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:49.477543 20366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:49.482044 20366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:49.482312 20366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:49.500279 20366 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:38703
I20260618 22:41:49.500262 20433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:38703 every 8 connection(s)
I20260618 22:41:49.501893 20366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:41:49.509001 20434 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:41:49.510820 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 20366
I20260618 22:41:49.511574 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:44055
--local_ip_for_outbound_sockets=127.5.213.193
--tserver_master_addrs=127.5.213.254:38703
--webserver_port=36379
--webserver_interface=127.5.213.193
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:41:49.512437 20434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b: Bootstrap starting.
I20260618 22:41:49.532034 20434 log.cc:824] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:41:49.544466 20434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:41:49.544986 20434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b: Bootstrap complete.
I20260618 22:41:49.554023 20434 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5c527d935d44cebcdda67d63b35d3b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 38703 } }
I20260618 22:41:49.554916 20434 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb5c527d935d44cebcdda67d63b35d3b, State: Initialized, Role: FOLLOWER
I20260618 22:41:49.555575 20434 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5c527d935d44cebcdda67d63b35d3b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 38703 } }
I20260618 22:41:49.555809 20434 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:41:49.555920 20434 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:41:49.556073 20434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 1 FOLLOWER]: Advancing to term 2
I20260618 22:41:49.560300 20434 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5c527d935d44cebcdda67d63b35d3b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 38703 } }
I20260618 22:41:49.560823 20434 leader_election.cc:302] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb5c527d935d44cebcdda67d63b35d3b; no voters:
I20260618 22:41:49.561444 20434 leader_election.cc:288] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260618 22:41:49.561579 20438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 2 FOLLOWER]: Leader election won for term 2
I20260618 22:41:49.562738 20438 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 2 LEADER]: Becoming Leader. State: Replica: fb5c527d935d44cebcdda67d63b35d3b, State: Running, Role: LEADER
I20260618 22:41:49.563287 20438 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5c527d935d44cebcdda67d63b35d3b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 38703 } }
I20260618 22:41:49.563802 20434 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:41:49.565716 20439 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fb5c527d935d44cebcdda67d63b35d3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5c527d935d44cebcdda67d63b35d3b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 38703 } } }
I20260618 22:41:49.566087 20439 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [sys.catalog]: This master's current role is: LEADER
I20260618 22:41:49.566350 20440 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb5c527d935d44cebcdda67d63b35d3b. Latest consensus state: current_term: 2 leader_uuid: "fb5c527d935d44cebcdda67d63b35d3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5c527d935d44cebcdda67d63b35d3b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 38703 } } }
I20260618 22:41:49.567058 20440 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [sys.catalog]: This master's current role is: LEADER
I20260618 22:41:49.569952 20443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:41:49.578542 20443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:41:49.580279 20443 catalog_manager.cc:1259] Loaded cluster ID: 99462a0954894858b735fe598ffdc5a4
I20260618 22:41:49.580442 20443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:41:49.583711 20443 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260618 22:41:49.600306 20443 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260618 22:41:49.609681 20443 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:41:49.611479 20443 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b: Loaded TSK: 0
I20260618 22:41:49.613061 20443 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 22:41:49.846252 20436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:49.846530 20436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:49.846666 20436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:49.853916 20436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:49.854023 20436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:49.854168 20436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:41:49.864137 20436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:44055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=36379
--tserver_master_addrs=127.5.213.254:38703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:49.865439 20436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:49.866791 20436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:49.881587 20462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:49.881711 20464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:49.882159 20461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:49.882339 20436 server_base.cc:1034] running on GCE node
I20260618 22:41:49.883047 20436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:49.884043 20436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:49.885265 20436 hybrid_clock.cc:648] HybridClock initialized: now 1781822509885213 us; error 35 us; skew 500 ppm
I20260618 22:41:49.885605 20436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:49 dist-test-slave-lnj3 krb5kdc[19814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822509, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:49.889043 20436 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:41:49.890588 20436 webserver.cc:460] Webserver started at http://127.5.213.193:36379/ using document root <none> and password file <none>
I20260618 22:41:49.891321 20436 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:49.891450 20436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:41:49.897192 20436 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260618 22:41:49.900180 20471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:49.901733 20436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:41:49.901943 20436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "efcc780097604e108f1bf64589be096a"
format_stamp: "Formatted at 2026-06-18 22:41:47 on dist-test-slave-lnj3"
server_key: "40e8f261e70aaa6ad33ad8219944774e"
server_key_iv: "c9c7dc8d1321ff0ba7db6fdac4b508ea"
server_key_version: "encryptionkey@0"
I20260618 22:41:49.902989 20436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:49.923573 20436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:49.927786 20436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:49.928037 20436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:49.929035 20436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:41:49.930426 20436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:41:49.930517 20436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:49.930649 20436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:41:49.930742 20436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260618 22:41:49.961537 20436 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:44055
I20260618 22:41:49.961594 20583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:44055 every 8 connection(s)
I20260618 22:41:49.963013 20436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:41:49.973445 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 20436
I20260618 22:41:49.974069 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:32851
--local_ip_for_outbound_sockets=127.5.213.194
--tserver_master_addrs=127.5.213.254:38703
--webserver_port=37881
--webserver_interface=127.5.213.194
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:41:49 dist-test-slave-lnj3 krb5kdc[19814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822509, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:41:49.987918 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:49.965995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:45177)
0618 22:41:49.966791 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:49.966842 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:49.968504 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:49.969828 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:49.969836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:49.970333 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:49.970957 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:49.970976 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:49.972651 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:49.972655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:49.973908 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:49.973920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:49.975295 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:49.975301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:49.975485 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:49.976104 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:49.976130 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:49.978397 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:49.982770 (+ 4373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:49.982780 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:49.982796 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:49.983540 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:49.984097 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:49.984102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:49.984105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:49.984323 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:49.985236 (+ 913us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:49.985244 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:49.985680 (+ 436us) client_negotiation.cc:769] Sending connection context
0618 22:41:49.986022 (+ 342us) client_negotiation.cc:240] Negotiation successful
0618 22:41:49.986516 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":314,"threads_started":1}
I20260618 22:41:49.988561 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:49.965836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:45177 (local address 127.5.213.254:38703)
0618 22:41:49.966395 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:49.966407 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:49.966958 (+ 551us) server_negotiation.cc:407] Connection header received
0618 22:41:49.969011 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:49.969024 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:49.969378 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:49.969692 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:49.971187 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:49.972524 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:49.974042 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:49.975078 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:49.975273 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:49.978707 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:49.978717 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:49.978737 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:49.978787 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:49.982570 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:49.983763 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:49.983772 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:49.983781 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:49.983974 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:49.984439 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:49.984444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:49.984446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:49.985043 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:49.985220 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:49.986957 (+ 1737us) server_negotiation.cc:299] Negotiation successful
0618 22:41:49.987393 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":262,"threads_started":1}
I20260618 22:41:49.991246 20584 heartbeater.cc:347] Connected to a master server at 127.5.213.254:38703
I20260618 22:41:49.991611 20584 heartbeater.cc:464] Registering TS with master...
I20260618 22:41:49.992527 20584 heartbeater.cc:511] Master 127.5.213.254:38703 requested a full tablet report, sending...
I20260618 22:41:49.994850 20399 ts_manager.cc:194] Registered new tserver with Master: efcc780097604e108f1bf64589be096a (127.5.213.193:44055)
I20260618 22:41:49.996363 20399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:45177
W20260618 22:41:50.324326 20588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:50.324712 20588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:50.324872 20588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:50.337095 20588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:50.337256 20588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:50.337448 20588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:41:50.353886 20588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:32851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=37881
--tserver_master_addrs=127.5.213.254:38703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:50.355760 20588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:50.357621 20588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:50.373566 20594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:50.373950 20596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:50.373988 20593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:50.374401 20588 server_base.cc:1034] running on GCE node
I20260618 22:41:50.375025 20588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:50.375944 20588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:50.377132 20588 hybrid_clock.cc:648] HybridClock initialized: now 1781822510377070 us; error 58 us; skew 500 ppm
I20260618 22:41:50.377447 20588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:50 dist-test-slave-lnj3 krb5kdc[19814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822510, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:50.380411 20588 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:41:50.381829 20588 webserver.cc:460] Webserver started at http://127.5.213.194:37881/ using document root <none> and password file <none>
I20260618 22:41:50.382498 20588 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:50.382617 20588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:41:50.388159 20588 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260618 22:41:50.391177 20603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:50.392695 20588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260618 22:41:50.392854 20588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "eb1bc427c6104d1aa18c574cfcbeb05b"
format_stamp: "Formatted at 2026-06-18 22:41:47 on dist-test-slave-lnj3"
server_key: "351753726aae08919cd03dc30c7d0ee5"
server_key_iv: "c6e0403ac895f5689d29255b9e6cfc76"
server_key_version: "encryptionkey@0"
I20260618 22:41:50.393834 20588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:50.411178 20588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:50.415166 20588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:50.415413 20588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:50.416412 20588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:41:50.417694 20588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:41:50.417789 20588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:50.417894 20588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:41:50.417959 20588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:50.450064 20588 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:32851
I20260618 22:41:50.450103 20715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:32851 every 8 connection(s)
I20260618 22:41:50.451704 20588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:41:50.459903 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 20588
I20260618 22:41:50.460641 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:44899
--local_ip_for_outbound_sockets=127.5.213.195
--tserver_master_addrs=127.5.213.254:38703
--webserver_port=43757
--webserver_interface=127.5.213.195
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:41:50 dist-test-slave-lnj3 krb5kdc[19814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822510, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:41:50.472041 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:50.454287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:54749 (local address 127.5.213.254:38703)
0618 22:41:50.454443 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:50.454448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:50.455952 (+ 1504us) server_negotiation.cc:407] Connection header received
0618 22:41:50.457812 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:50.457818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:50.457899 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:50.457985 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:50.459637 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.460416 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.462034 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.462449 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.462558 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:50.466024 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:50.466029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:50.466032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:50.466066 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:50.468453 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:50.469278 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:50.469285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:50.469289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:50.469452 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:50.469939 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:50.469945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:50.469948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:50.470314 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:50.470501 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:50.471341 (+ 840us) server_negotiation.cc:299] Negotiation successful
0618 22:41:50.471625 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:41:50.473356 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:50.455012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.194:54749)
0618 22:41:50.455803 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:50.455840 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:50.457549 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:50.458238 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:50.458249 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:50.458855 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:50.459432 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:50.459454 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.460574 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.460577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:50.461890 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:50.461903 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.462682 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.462691 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:50.462934 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:50.463592 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:50.463623 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:50.465827 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:50.468581 (+ 2754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:50.468590 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:50.468607 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:50.469128 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:50.469566 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:50.469570 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:50.469573 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:50.469820 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:50.470463 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:50.470471 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:50.470925 (+ 454us) client_negotiation.cc:769] Sending connection context
0618 22:41:50.471215 (+ 290us) client_negotiation.cc:240] Negotiation successful
0618 22:41:50.471711 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":290,"threads_started":1}
I20260618 22:41:50.474906 20716 heartbeater.cc:347] Connected to a master server at 127.5.213.254:38703
I20260618 22:41:50.475203 20716 heartbeater.cc:464] Registering TS with master...
I20260618 22:41:50.476027 20716 heartbeater.cc:511] Master 127.5.213.254:38703 requested a full tablet report, sending...
I20260618 22:41:50.477698 20399 ts_manager.cc:194] Registered new tserver with Master: eb1bc427c6104d1aa18c574cfcbeb05b (127.5.213.194:32851)
I20260618 22:41:50.478904 20399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:54749
W20260618 22:41:50.779139 20719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:50.779426 20719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:50.779562 20719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:50.786783 20719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:50.786901 20719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:50.787051 20719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:41:50.797134 20719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:44899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=43757
--tserver_master_addrs=127.5.213.254:38703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:50.798488 20719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:50.799805 20719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:50.812305 20725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:50.812383 20724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:50.812331 20727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:50.812973 20719 server_base.cc:1034] running on GCE node
I20260618 22:41:50.813578 20719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:50.814543 20719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:50.815778 20719 hybrid_clock.cc:648] HybridClock initialized: now 1781822510815704 us; error 38 us; skew 500 ppm
I20260618 22:41:50.816076 20719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:50 dist-test-slave-lnj3 krb5kdc[19814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822510, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:50.819099 20719 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:41:50.820475 20719 webserver.cc:460] Webserver started at http://127.5.213.195:43757/ using document root <none> and password file <none>
I20260618 22:41:50.821195 20719 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:50.821322 20719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:41:50.826596 20719 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260618 22:41:50.829463 20734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:50.830926 20719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:41:50.831076 20719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "4bdb44283a794928bd76eddf3443eaa4"
format_stamp: "Formatted at 2026-06-18 22:41:48 on dist-test-slave-lnj3"
server_key: "dce57b15eb94384fc83048c5f1c6278b"
server_key_iv: "d4f2301607490b78c8aa121e25aa8dd8"
server_key_version: "encryptionkey@0"
I20260618 22:41:50.832173 20719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:50.855158 20719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:50.859117 20719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:50.859362 20719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:50.860344 20719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:41:50.861635 20719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:41:50.861738 20719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:50.861845 20719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:41:50.861912 20719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:50.891817 20719 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:44899
I20260618 22:41:50.891882 20846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:44899 every 8 connection(s)
I20260618 22:41:50.893582 20719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:41:50.896777 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 20719
Jun 18 22:41:50 dist-test-slave-lnj3 krb5kdc[19814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822510, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:41:50.915542 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:50.896302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:33381 (local address 127.5.213.254:38703)
0618 22:41:50.896556 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:50.896563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:50.897841 (+ 1278us) server_negotiation.cc:407] Connection header received
0618 22:41:50.899837 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:50.899842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:50.899929 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:50.900028 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:50.901812 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.903073 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.904782 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.905253 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.905358 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:50.909169 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:50.909174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:50.909178 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:50.909208 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:50.912423 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:50.913099 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:50.913105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:50.913108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:50.913255 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:50.913745 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:50.913749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:50.913751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:50.914091 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:50.914242 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:50.915021 (+ 779us) server_negotiation.cc:299] Negotiation successful
0618 22:41:50.915206 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260618 22:41:50.916113 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:50.897629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.5.213.254:38703)
0618 22:41:50.903235 (+ 5606us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:50.903240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:50.903257 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:41:50.903306 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:50.903310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:50.903378 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:50.903478 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:50.904946 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.905748 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.907186 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.907684 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.907808 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:50.910392 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:50.910398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:50.910403 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:50.910455 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:50.913282 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:50.913972 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:50.913977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:50.913981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:50.914101 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:50.914476 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:50.914479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:50.914481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:50.914815 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:50.915038 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:50.915335 (+ 297us) server_negotiation.cc:299] Negotiation successful
0618 22:41:50.915525 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5494,"spinlock_wait_cycles":8576,"thread_start_us":5375,"threads_started":1}
I20260618 22:41:50.916751 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:50.896784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.195:33381)
0618 22:41:50.897606 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:50.897717 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:50.899559 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:50.900472 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:50.900484 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:50.901032 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:50.901622 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:50.901640 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.903272 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.903278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:50.904633 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:50.904645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.905466 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.905474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:50.905716 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:50.906348 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:50.906377 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:50.908919 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:50.912545 (+ 3626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:50.912553 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:50.912566 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:50.912981 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:50.913354 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:50.913359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:50.913362 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:50.913601 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:50.914227 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:50.914235 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:50.914658 (+ 423us) client_negotiation.cc:769] Sending connection context
0618 22:41:50.914931 (+ 273us) client_negotiation.cc:240] Negotiation successful
0618 22:41:50.915480 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":303,"threads_started":1}
I20260618 22:41:50.918114 20847 heartbeater.cc:347] Connected to a master server at 127.5.213.254:38703
I20260618 22:41:50.918419 20847 heartbeater.cc:464] Registering TS with master...
I20260618 22:41:50.919226 20847 heartbeater.cc:511] Master 127.5.213.254:38703 requested a full tablet report, sending...
I20260618 22:41:50.920598 20398 ts_manager.cc:194] Registered new tserver with Master: 4bdb44283a794928bd76eddf3443eaa4 (127.5.213.195:44899)
I20260618 22:41:50.921468 20398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:33381
I20260618 22:41:50.931354 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:41:50.938480 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:50.932267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45504 (local address 127.5.213.254:38703)
0618 22:41:50.932425 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:50.932429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:50.932895 (+ 466us) server_negotiation.cc:407] Connection header received
0618 22:41:50.933113 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:50.933117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:50.933271 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:50.933402 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:41:50.933803 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.934512 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.936195 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.936722 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.936827 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:50.937469 (+ 642us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:41:50.937983 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:41:50.938107 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:50.938206 (+ 99us) server_negotiation.cc:299] Negotiation successful
0618 22:41:50.938282 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:41:50.939488 20398 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:41:50.946728 20398 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:41:50.983089 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:50.958397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:44055 (local address 127.0.0.1:47296)
0618 22:41:50.973994 (+ 15597us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:50.974014 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:50.974135 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:50.976222 (+ 2087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:50.976227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:50.976263 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:41:50.976754 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:50.976766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.978516 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.978520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:50.980493 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:50.980510 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.982016 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.982022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:50.982211 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:50.982254 (+ 43us) client_negotiation.cc:769] Sending connection context
0618 22:41:50.982389 (+ 135us) client_negotiation.cc:240] Negotiation successful
0618 22:41:50.982589 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15465,"mutex_wait_us":70,"thread_start_us":16685,"threads_started":1}
I20260618 22:41:50.983387 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:50.958514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.5.213.194:32851)
0618 22:41:50.974934 (+ 16420us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:50.974942 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:50.975037 (+ 95us) server_negotiation.cc:407] Connection header received
0618 22:41:50.975156 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:50.975164 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:50.975443 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:50.975652 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:41:50.976407 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.977915 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.980786 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.982190 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.982319 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:50.982540 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:50.982800 (+ 260us) server_negotiation.cc:299] Negotiation successful
0618 22:41:50.982968 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16288,"thread_start_us":16240,"threads_started":1}
I20260618 22:41:50.983424 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:50.958336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:32851 (local address 127.0.0.1:40018)
0618 22:41:50.970363 (+ 12027us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:50.970412 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:50.970588 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:50.975687 (+ 5099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:50.975696 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:50.975727 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:41:50.976147 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:50.976163 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.978066 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.978072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:50.980555 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:50.980574 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.982417 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.982422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:50.982593 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:50.982650 (+ 57us) client_negotiation.cc:769] Sending connection context
0618 22:41:50.982739 (+ 89us) client_negotiation.cc:240] Negotiation successful
0618 22:41:50.982877 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11736,"mutex_wait_us":97,"thread_start_us":12460,"threads_started":1}
I20260618 22:41:50.983558 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:50.959536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:44899 (local address 127.0.0.1:54244)
0618 22:41:50.975312 (+ 15776us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:50.975328 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:50.975518 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:50.976123 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:50.976128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:50.976150 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:41:50.976540 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:50.976552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.977839 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.977844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:50.980586 (+ 2742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:50.980596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.982713 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.982719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:50.982888 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:50.982925 (+ 37us) client_negotiation.cc:769] Sending connection context
0618 22:41:50.983017 (+ 92us) client_negotiation.cc:240] Negotiation successful
0618 22:41:50.983153 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15672,"mutex_wait_us":186,"thread_start_us":14184,"threads_started":1}
I20260618 22:41:50.984062 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:50.958950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47296 (local address 127.5.213.193:44055)
0618 22:41:50.975645 (+ 16695us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:50.975652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:50.975675 (+ 23us) server_negotiation.cc:407] Connection header received
0618 22:41:50.975755 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:50.975762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:50.976008 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:50.976949 (+ 941us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:41:50.977161 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.978399 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.980685 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.981830 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.981951 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:50.982169 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:50.983580 (+ 1411us) server_negotiation.cc:299] Negotiation successful
0618 22:41:50.983766 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16538,"thread_start_us":16418,"threads_started":1}
I20260618 22:41:50.984254 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:50.959123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54244 (local address 127.5.213.195:44899)
0618 22:41:50.974712 (+ 15589us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:50.974721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:50.975423 (+ 702us) server_negotiation.cc:407] Connection header received
0618 22:41:50.975716 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:50.975724 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:50.975950 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:50.976104 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:41:50.976715 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.977699 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.980749 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:50.982191 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:50.982325 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:50.982540 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:50.983811 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0618 22:41:50.983956 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15394,"thread_start_us":15242,"threads_started":1}
I20260618 22:41:50.987231 20519 tablet_service.cc:1467] Processing CreateTablet for tablet 081b7ff09ccf432fb6510c5c8f490518 (DEFAULT_TABLE table=test-table [id=d6204fce82fd46b7ac1df1e5ba357254]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:41:50.987443 20651 tablet_service.cc:1467] Processing CreateTablet for tablet 081b7ff09ccf432fb6510c5c8f490518 (DEFAULT_TABLE table=test-table [id=d6204fce82fd46b7ac1df1e5ba357254]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:41:50.987443 20782 tablet_service.cc:1467] Processing CreateTablet for tablet 081b7ff09ccf432fb6510c5c8f490518 (DEFAULT_TABLE table=test-table [id=d6204fce82fd46b7ac1df1e5ba357254]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:41:50.988758 20519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081b7ff09ccf432fb6510c5c8f490518. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:41:50.988763 20782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081b7ff09ccf432fb6510c5c8f490518. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:41:50.988787 20651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081b7ff09ccf432fb6510c5c8f490518. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:41:50.999805 20584 heartbeater.cc:503] Master 127.5.213.254:38703 was elected leader, sending a full tablet report...
I20260618 22:41:51.004313 20863 tablet_bootstrap.cc:492] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a: Bootstrap starting.
I20260618 22:41:51.005093 20864 tablet_bootstrap.cc:492] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b: Bootstrap starting.
I20260618 22:41:51.009189 20863 tablet_bootstrap.cc:654] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a: Neither blocks nor log segments found. Creating new log.
I20260618 22:41:51.009650 20865 tablet_bootstrap.cc:492] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4: Bootstrap starting.
I20260618 22:41:51.010090 20864 tablet_bootstrap.cc:654] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b: Neither blocks nor log segments found. Creating new log.
I20260618 22:41:51.010488 20863 log.cc:824] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:41:51.011343 20864 log.cc:824] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:41:51.012562 20865 tablet_bootstrap.cc:654] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4: Neither blocks nor log segments found. Creating new log.
I20260618 22:41:51.013439 20865 log.cc:824] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4: Log is configured to *not* fsync() on all Append() calls
I20260618 22:41:51.014009 20863 tablet_bootstrap.cc:492] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a: No bootstrap required, opened a new log
I20260618 22:41:51.014353 20863 ts_tablet_manager.cc:1397] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260618 22:41:51.016079 20865 tablet_bootstrap.cc:492] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4: No bootstrap required, opened a new log
I20260618 22:41:51.016330 20865 ts_tablet_manager.cc:1397] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260618 22:41:51.017071 20864 tablet_bootstrap.cc:492] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b: No bootstrap required, opened a new log
I20260618 22:41:51.017321 20864 ts_tablet_manager.cc:1397] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260618 22:41:51.023733 20865 raft_consensus.cc:348] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:51.024106 20865 raft_consensus.cc:374] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:41:51.024204 20865 raft_consensus.cc:729] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bdb44283a794928bd76eddf3443eaa4, State: Initialized, Role: FOLLOWER
I20260618 22:41:51.024807 20865 consensus_queue.cc:260] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:51.024703 20863 raft_consensus.cc:348] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:51.025125 20863 raft_consensus.cc:374] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:41:51.025218 20863 raft_consensus.cc:729] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efcc780097604e108f1bf64589be096a, State: Initialized, Role: FOLLOWER
I20260618 22:41:51.025842 20863 consensus_queue.cc:260] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:51.027421 20864 raft_consensus.cc:348] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:51.027738 20863 ts_tablet_manager.cc:1428] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260618 22:41:51.027853 20864 raft_consensus.cc:374] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:41:51.027938 20864 raft_consensus.cc:729] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb1bc427c6104d1aa18c574cfcbeb05b, State: Initialized, Role: FOLLOWER
I20260618 22:41:51.029121 20847 heartbeater.cc:503] Master 127.5.213.254:38703 was elected leader, sending a full tablet report...
I20260618 22:41:51.029817 20865 ts_tablet_manager.cc:1428] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
I20260618 22:41:51.028726 20864 consensus_queue.cc:260] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:51.031158 20716 heartbeater.cc:503] Master 127.5.213.254:38703 was elected leader, sending a full tablet report...
I20260618 22:41:51.031653 20864 ts_tablet_manager.cc:1428] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260618 22:41:51.112361 20869 raft_consensus.cc:482] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:41:51.112620 20869 raft_consensus.cc:504] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:51.114185 20869 leader_election.cc:288] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb1bc427c6104d1aa18c574cfcbeb05b (127.5.213.194:32851), efcc780097604e108f1bf64589be096a (127.5.213.193:44055)
I20260618 22:41:51.120363 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:51.114682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:45105 (local address 127.5.213.193:44055)
0618 22:41:51.114879 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:51.114885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:51.114905 (+ 20us) server_negotiation.cc:407] Connection header received
0618 22:41:51.115153 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:51.115159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:51.115239 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:51.115344 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:41:51.115729 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:51.116682 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:51.118447 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:51.119492 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:51.119613 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:51.119668 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:51.120010 (+ 342us) server_negotiation.cc:299] Negotiation successful
0618 22:41:51.120102 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260618 22:41:51.120432 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:51.114441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:44055 (local address 127.5.213.195:45105)
0618 22:41:51.114826 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:51.114849 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:51.114991 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:51.115374 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:51.115378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:51.115400 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:41:51.115592 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:51.115604 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:51.116824 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:51.116829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:51.118288 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:51.118300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:51.119641 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:51.119648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:51.119860 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:51.119896 (+ 36us) client_negotiation.cc:769] Sending connection context
0618 22:41:51.119996 (+ 100us) client_negotiation.cc:240] Negotiation successful
0618 22:41:51.120128 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260618 22:41:51.121001 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:51.114689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:32857 (local address 127.5.213.194:32851)
0618 22:41:51.114879 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:51.114885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:51.115497 (+ 612us) server_negotiation.cc:407] Connection header received
0618 22:41:51.115640 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:51.115645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:51.115723 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:51.115793 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:41:51.116391 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:51.117241 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:51.119124 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:51.120065 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:51.120194 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:51.120272 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:51.120643 (+ 371us) server_negotiation.cc:299] Negotiation successful
0618 22:41:51.120767 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:41:51.121629 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:51.114507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:32851 (local address 127.5.213.195:32857)
0618 22:41:51.115382 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:51.115400 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:51.115518 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:51.115833 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:51.115838 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:51.115862 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:41:51.116204 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:51.116215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:51.117398 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:51.117402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:51.118974 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:51.118988 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:51.120277 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:51.120283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:51.120459 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:51.120490 (+ 31us) client_negotiation.cc:769] Sending connection context
0618 22:41:51.121169 (+ 679us) client_negotiation.cc:240] Negotiation successful
0618 22:41:51.121328 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":555,"threads_started":1}
I20260618 22:41:51.121619 20539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081b7ff09ccf432fb6510c5c8f490518" candidate_uuid: "4bdb44283a794928bd76eddf3443eaa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efcc780097604e108f1bf64589be096a" is_pre_election: true
I20260618 22:41:51.122247 20539 raft_consensus.cc:2454] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bdb44283a794928bd76eddf3443eaa4 in term 0.
I20260618 22:41:51.122706 20671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081b7ff09ccf432fb6510c5c8f490518" candidate_uuid: "4bdb44283a794928bd76eddf3443eaa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" is_pre_election: true
I20260618 22:41:51.123027 20737 leader_election.cc:302] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bdb44283a794928bd76eddf3443eaa4, efcc780097604e108f1bf64589be096a; no voters:
I20260618 22:41:51.123219 20671 raft_consensus.cc:2454] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bdb44283a794928bd76eddf3443eaa4 in term 0.
I20260618 22:41:51.123452 20869 raft_consensus.cc:2781] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:41:51.123588 20869 raft_consensus.cc:482] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:41:51.123739 20869 raft_consensus.cc:3037] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:41:51.125954 20869 raft_consensus.cc:504] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:51.126652 20869 leader_election.cc:288] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [CANDIDATE]: Term 1 election: Requested vote from peers eb1bc427c6104d1aa18c574cfcbeb05b (127.5.213.194:32851), efcc780097604e108f1bf64589be096a (127.5.213.193:44055)
I20260618 22:41:51.127085 20671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081b7ff09ccf432fb6510c5c8f490518" candidate_uuid: "4bdb44283a794928bd76eddf3443eaa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b"
I20260618 22:41:51.127310 20671 raft_consensus.cc:3037] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:41:51.127411 20539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081b7ff09ccf432fb6510c5c8f490518" candidate_uuid: "4bdb44283a794928bd76eddf3443eaa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efcc780097604e108f1bf64589be096a"
I20260618 22:41:51.127614 20539 raft_consensus.cc:3037] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:41:51.130493 20671 raft_consensus.cc:2454] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bdb44283a794928bd76eddf3443eaa4 in term 1.
I20260618 22:41:51.130563 20539 raft_consensus.cc:2454] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bdb44283a794928bd76eddf3443eaa4 in term 1.
I20260618 22:41:51.130963 20735 leader_election.cc:302] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bdb44283a794928bd76eddf3443eaa4, eb1bc427c6104d1aa18c574cfcbeb05b; no voters:
I20260618 22:41:51.131232 20869 raft_consensus.cc:2781] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:41:51.131649 20869 raft_consensus.cc:686] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [term 1 LEADER]: Becoming Leader. State: Replica: 4bdb44283a794928bd76eddf3443eaa4, State: Running, Role: LEADER
I20260618 22:41:51.132107 20869 consensus_queue.cc:237] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:51.136842 20398 catalog_manager.cc:5495] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bdb44283a794928bd76eddf3443eaa4 (127.5.213.195). New cstate: current_term: 1 leader_uuid: "4bdb44283a794928bd76eddf3443eaa4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } health_report { overall_health: HEALTHY } } }
W20260618 22:41:51.145146 20848 tablet.cc:2369] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:41:51.160372 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:51.155140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54254 (local address 127.5.213.195:44899)
0618 22:41:51.155303 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:51.155307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:51.155391 (+ 84us) server_negotiation.cc:407] Connection header received
0618 22:41:51.155569 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:51.155573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:51.155675 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:51.155800 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:41:51.156157 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:51.156921 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:51.158336 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:51.158813 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:51.158905 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:51.159389 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:41:51.159922 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:41:51.160029 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:51.160105 (+ 76us) server_negotiation.cc:299] Negotiation successful
0618 22:41:51.160177 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:41:51.168196 20539 raft_consensus.cc:1261] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Refusing update from remote peer 4bdb44283a794928bd76eddf3443eaa4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:41:51.168201 20671 raft_consensus.cc:1261] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b [term 1 FOLLOWER]: Refusing update from remote peer 4bdb44283a794928bd76eddf3443eaa4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:41:51.169160 20873 consensus_queue.cc:1035] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:41:51.169497 20869 consensus_queue.cc:1035] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:41:51.179400 20878 mvcc.cc:205] Tried to move back new op lower bound from 7298345005735727104 to 7298345005600989184. Current Snapshot: MvccSnapshot[applied={T|T < 7298345005735727104}]
I20260618 22:41:51.179425 20876 mvcc.cc:205] Tried to move back new op lower bound from 7298345005735727104 to 7298345005600989184. Current Snapshot: MvccSnapshot[applied={T|T < 7298345005735727104}]
I20260618 22:41:51.187086 20877 mvcc.cc:205] Tried to move back new op lower bound from 7298345005735727104 to 7298345005600989184. Current Snapshot: MvccSnapshot[applied={T|T < 7298345005735727104}]
I20260618 22:41:51.196272 20398 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:41:51.196520 20398 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:41:51.199997 20398 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b: Sending DeleteTablet for 3 replicas of tablet 081b7ff09ccf432fb6510c5c8f490518
I20260618 22:41:51.201069 20651 tablet_service.cc:1514] Processing DeleteTablet for tablet 081b7ff09ccf432fb6510c5c8f490518 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:41:51 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:40018
I20260618 22:41:51.201069 20519 tablet_service.cc:1514] Processing DeleteTablet for tablet 081b7ff09ccf432fb6510c5c8f490518 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:41:51 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:47296
I20260618 22:41:51.201114 20782 tablet_service.cc:1514] Processing DeleteTablet for tablet 081b7ff09ccf432fb6510c5c8f490518 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:41:51 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:54244
I20260618 22:41:51.202390 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 20436
I20260618 22:41:51.217705 20888 tablet_replica.cc:330] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4: stopping tablet replica
I20260618 22:41:51.218278 20888 raft_consensus.cc:2229] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:41:51.219056 20888 raft_consensus.cc:2258] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:41:51.221268 20888 ts_tablet_manager.cc:1905] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:41:51.223419 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:51.203969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47298 (local address 127.5.213.193:44055)
0618 22:41:51.204154 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:51.204161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:51.204180 (+ 19us) server_negotiation.cc:407] Connection header received
0618 22:41:51.204241 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:51.204246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:51.204324 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:51.204418 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:51.206563 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:51.207893 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:51.210425 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:51.211178 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:51.211295 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:51.215121 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:51.215141 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:51.215163 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:51.215212 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:51.219206 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:51.220268 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:51.220277 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:51.220280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:51.220424 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:51.220904 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:51.220909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:51.220913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:51.221402 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:51.221667 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:51.222840 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0618 22:41:51.223049 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260618 22:41:51.224092 20579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:41:51.228755 20888 ts_tablet_manager.cc:1918] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:41:51.228976 20888 log.cc:1196] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/081b7ff09ccf432fb6510c5c8f490518
I20260618 22:41:51.229600 20888 ts_tablet_manager.cc:1939] T 081b7ff09ccf432fb6510c5c8f490518 P 4bdb44283a794928bd76eddf3443eaa4: Deleting consensus metadata
I20260618 22:41:51.231477 20384 catalog_manager.cc:4841] TS 4bdb44283a794928bd76eddf3443eaa4 (127.5.213.195:44899): tablet 081b7ff09ccf432fb6510c5c8f490518 (table test-table [id=d6204fce82fd46b7ac1df1e5ba357254]) successfully deleted
I20260618 22:41:51.244489 20887 tablet_replica.cc:330] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b: stopping tablet replica
I20260618 22:41:51.245057 20887 raft_consensus.cc:2229] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:41:51.245584 20887 raft_consensus.cc:2258] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:41:51.247767 20887 ts_tablet_manager.cc:1905] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:41:51.254812 20887 ts_tablet_manager.cc:1918] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:41:51.254972 20887 log.cc:1196] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/081b7ff09ccf432fb6510c5c8f490518
I20260618 22:41:51.255491 20887 ts_tablet_manager.cc:1939] T 081b7ff09ccf432fb6510c5c8f490518 P eb1bc427c6104d1aa18c574cfcbeb05b: Deleting consensus metadata
I20260618 22:41:51.257364 20383 catalog_manager.cc:4841] TS eb1bc427c6104d1aa18c574cfcbeb05b (127.5.213.194:32851): tablet 081b7ff09ccf432fb6510c5c8f490518 (table test-table [id=d6204fce82fd46b7ac1df1e5ba357254]) successfully deleted
I20260618 22:41:51.338376 20886 tablet_replica.cc:330] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a: stopping tablet replica
I20260618 22:41:51.338919 20886 raft_consensus.cc:2229] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:41:51.339368 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 20436
I20260618 22:41:51.339496 20886 raft_consensus.cc:2258] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:41:51.354080 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 20588
W20260618 22:41:51.354169 20385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.213.193:44055 (error 108)
I20260618 22:41:51.356355 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:51.355859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:44055 (local address 127.0.0.1:47310)
0618 22:41:51.356038 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:51.356155 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260618 22:41:51.356613 20385 catalog_manager.cc:4584] TS efcc780097604e108f1bf64589be096a (127.5.213.193:44055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 081b7ff09ccf432fb6510c5c8f490518: Network error: Client connection negotiation failed: client connection to 127.5.213.193:44055: connect: Connection refused (error 111)
I20260618 22:41:51.368247 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:51.355575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.5.213.194:32851)
0618 22:41:51.355741 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:51.355745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:51.355760 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:41:51.355811 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:51.355814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:51.355881 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:51.355967 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:51.357637 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:51.358486 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:51.359823 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:51.360224 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:51.360336 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:51.362782 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:51.362797 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:51.362811 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:51.362844 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:51.365603 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:51.366265 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:51.366277 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:51.366279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:51.366404 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:51.366774 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:51.366777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:51.366779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:51.367136 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:51.367429 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:51.367720 (+ 291us) server_negotiation.cc:299] Negotiation successful
0618 22:41:51.367926 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:41:51.368878 20711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:41:51.410518 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:51.409888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:44055 (local address 127.0.0.1:47318)
0618 22:41:51.410116 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:51.410255 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260618 22:41:51.411731 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:51.411306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:44055 (local address 127.0.0.1:47334)
0618 22:41:51.411464 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:51.411559 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260618 22:41:51.457182 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 20588
I20260618 22:41:51.470831 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 20719
I20260618 22:41:51.472786 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:51.472256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:44055 (local address 127.0.0.1:47338)
0618 22:41:51.472484 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:51.472577 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260618 22:41:51.473902 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:51.473513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:44055 (local address 127.0.0.1:47348)
0618 22:41:51.473672 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:51.473759 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260618 22:41:51.486390 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:51.471718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54256 (local address 127.5.213.195:44899)
0618 22:41:51.471935 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:51.471940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:51.471959 (+ 19us) server_negotiation.cc:407] Connection header received
0618 22:41:51.472038 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:51.472044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:51.472129 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:51.472220 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:51.473893 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:51.474809 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:51.476654 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:51.477107 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:51.477199 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:51.480437 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:51.480452 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:51.480472 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:51.480508 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:51.483463 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:51.484319 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:51.484326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:51.484328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:51.484474 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:51.484871 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:51.484874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:51.484876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:51.485281 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:51.485520 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:51.485905 (+ 385us) server_negotiation.cc:299] Negotiation successful
0618 22:41:51.486061 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260618 22:41:51.487172 20842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:41:51.555173 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:51.554558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:44055 (local address 127.0.0.1:47364)
0618 22:41:51.554879 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:51.554992 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260618 22:41:51.556461 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:51.555989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:44055 (local address 127.0.0.1:47372)
0618 22:41:51.556198 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:51.556310 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260618 22:41:51.616499 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 20719
I20260618 22:41:51.631531 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 20366
I20260618 22:41:51.632606 20429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:41:51.744994 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 20366
I20260618 22:41:51.762174 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:38703
--webserver_interface=127.5.213.254
--webserver_port=38019
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:38703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:41:52.078080 20894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:52.078565 20894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:52.078740 20894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:52.086345 20894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:41:52.086470 20894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:52.086552 20894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:52.086658 20894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:41:52.086724 20894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:41:52.096849 20894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:38703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:38703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=38019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:52.098327 20894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:52.099737 20894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:52.111524 20899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:52.111559 20900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:52.111449 20902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:52.112624 20894 server_base.cc:1034] running on GCE node
I20260618 22:41:52.113400 20894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:52.114866 20894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:52.116113 20894 hybrid_clock.cc:648] HybridClock initialized: now 1781822512116059 us; error 40 us; skew 500 ppm
I20260618 22:41:52.116480 20894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:52 dist-test-slave-lnj3 krb5kdc[19814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822512, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:52.119506 20894 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:41:52.120844 20894 webserver.cc:460] Webserver started at http://127.5.213.254:38019/ using document root <none> and password file <none>
I20260618 22:41:52.121562 20894 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:52.121662 20894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:41:52.126977 20894 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260618 22:41:52.129711 20909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:52.131443 20894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:41:52.131611 20894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "fb5c527d935d44cebcdda67d63b35d3b"
format_stamp: "Formatted at 2026-06-18 22:41:46 on dist-test-slave-lnj3"
server_key: "9654ed53e4401b6a6a8b7a47884a8951"
server_key_iv: "fb717e899f44f72d6db74aa783434169"
server_key_version: "encryptionkey@0"
I20260618 22:41:52.132745 20894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:52.151274 20894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:52.155174 20894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:52.155386 20894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:52.172123 20894 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:38703
I20260618 22:41:52.172143 20960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:38703 every 8 connection(s)
I20260618 22:41:52.173537 20894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:41:52.180033 20961 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:41:52.181819 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 20894
I20260618 22:41:52.182601 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:44055
--local_ip_for_outbound_sockets=127.5.213.193
--tserver_master_addrs=127.5.213.254:38703
--webserver_port=36379
--webserver_interface=127.5.213.193
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:41:52.183287 20961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b: Bootstrap starting.
I20260618 22:41:52.208318 20961 log.cc:824] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:41:52.229151 20961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:41:52.229803 20961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b: Bootstrap complete.
I20260618 22:41:52.243022 20961 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5c527d935d44cebcdda67d63b35d3b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 38703 } }
I20260618 22:41:52.244246 20961 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb5c527d935d44cebcdda67d63b35d3b, State: Initialized, Role: FOLLOWER
I20260618 22:41:52.245007 20961 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5c527d935d44cebcdda67d63b35d3b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 38703 } }
I20260618 22:41:52.245312 20961 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:41:52.245441 20961 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:41:52.245652 20961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 2 FOLLOWER]: Advancing to term 3
I20260618 22:41:52.251375 20961 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5c527d935d44cebcdda67d63b35d3b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 38703 } }
I20260618 22:41:52.251967 20961 leader_election.cc:302] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb5c527d935d44cebcdda67d63b35d3b; no voters:
I20260618 22:41:52.252624 20961 leader_election.cc:288] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260618 22:41:52.252789 20965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 3 FOLLOWER]: Leader election won for term 3
I20260618 22:41:52.253827 20965 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [term 3 LEADER]: Becoming Leader. State: Replica: fb5c527d935d44cebcdda67d63b35d3b, State: Running, Role: LEADER
I20260618 22:41:52.254402 20965 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5c527d935d44cebcdda67d63b35d3b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 38703 } }
I20260618 22:41:52.254753 20961 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:41:52.256593 20967 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb5c527d935d44cebcdda67d63b35d3b. Latest consensus state: current_term: 3 leader_uuid: "fb5c527d935d44cebcdda67d63b35d3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5c527d935d44cebcdda67d63b35d3b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 38703 } } }
I20260618 22:41:52.256919 20967 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [sys.catalog]: This master's current role is: LEADER
I20260618 22:41:52.257390 20966 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fb5c527d935d44cebcdda67d63b35d3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5c527d935d44cebcdda67d63b35d3b" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 38703 } } }
I20260618 22:41:52.257642 20966 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5c527d935d44cebcdda67d63b35d3b [sys.catalog]: This master's current role is: LEADER
I20260618 22:41:52.259307 20970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:41:52.267663 20970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:41:52.269162 20970 catalog_manager.cc:1259] Loaded cluster ID: 99462a0954894858b735fe598ffdc5a4
I20260618 22:41:52.269315 20970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260618 22:41:52.271291 20970 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4e3536e9ad google::LogMessage::Fail() at ??:0
@ 0x7f4e3537188a google::LogMessage::SendToLog() at ??:0
@ 0x7f4e3536e4de google::LogMessage::Flush() at ??:0
@ 0x7f4e3536fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4e4ada1512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f4e4ad9e976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
I20260618 22:41:52.298193 20983 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f4e4ae26610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f4e4ae26439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4e37fba345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4e37fbd91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f4e37fbd799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4e37f98eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4e3b2726db start_thread at ??:0
@ 0x7f4e3420861f clone at ??:0
W20260618 22:41:52.553304 20963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:52.553589 20963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:52.553723 20963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:52.561088 20963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:52.561225 20963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:52.561390 20963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:41:52.571305 20963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:44055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=36379
--tserver_master_addrs=127.5.213.254:38703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:52.572636 20963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:52.573966 20963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:52.587203 21003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:52.587255 21002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:52.587456 21005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:52.588663 20963 server_base.cc:1034] running on GCE node
I20260618 22:41:52.589277 20963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:52.590273 20963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:52.591516 20963 hybrid_clock.cc:648] HybridClock initialized: now 1781822512591445 us; error 61 us; skew 500 ppm
I20260618 22:41:52.591856 20963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:52 dist-test-slave-lnj3 krb5kdc[19814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822512, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:52.595055 20963 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:41:52.596531 20963 webserver.cc:460] Webserver started at http://127.5.213.193:36379/ using document root <none> and password file <none>
I20260618 22:41:52.597316 20963 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:52.597450 20963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:41:52.602806 20963 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:41:52.605724 21012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:52.607209 20963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:41:52.607370 20963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "efcc780097604e108f1bf64589be096a"
format_stamp: "Formatted at 2026-06-18 22:41:47 on dist-test-slave-lnj3"
server_key: "40e8f261e70aaa6ad33ad8219944774e"
server_key_iv: "c9c7dc8d1321ff0ba7db6fdac4b508ea"
server_key_version: "encryptionkey@0"
I20260618 22:41:52.608386 20963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:52.625278 20963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:52.629240 20963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:52.629506 20963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:52.630537 20963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:41:52.632287 21019 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260618 22:41:52.638036 20963 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 22:41:52.638181 20963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260618 22:41:52.638311 20963 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 22:41:52.641549 20963 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 22:41:52.641631 20963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260618 22:41:52.642051 21019 tablet_bootstrap.cc:492] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a: Bootstrap starting.
I20260618 22:41:52.667810 21019 log.cc:824] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:41:52.679224 20963 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:44055
I20260618 22:41:52.679266 21126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:44055 every 8 connection(s)
I20260618 22:41:52.680461 20963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:41:52.680575 21019 tablet_bootstrap.cc:492] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:41:52.681093 21019 tablet_bootstrap.cc:492] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a: Bootstrap complete.
I20260618 22:41:52.682185 21019 ts_tablet_manager.cc:1397] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a: Time spent bootstrapping tablet: real 0.041s user 0.035s sys 0.004s
I20260618 22:41:52.686548 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:52.683635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:46683)
0618 22:41:52.684782 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:52.685123 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":453,"threads_started":1}
W20260618 22:41:52.687665 21127 heartbeater.cc:644] Failed to heartbeat to 127.5.213.254:38703 (0 consecutive failures): Network error: Failed to ping master at 127.5.213.254:38703: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
I20260618 22:41:52.688544 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:52.688193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:51351)
0618 22:41:52.688364 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:52.688440 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260618 22:41:52.689378 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 20963
I20260618 22:41:52.689474 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:52.689110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:36297)
0618 22:41:52.689256 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:52.689338 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260618 22:41:52.689724 21127 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 22:41:52.690099 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:32851
--local_ip_for_outbound_sockets=127.5.213.194
--tserver_master_addrs=127.5.213.254:38703
--webserver_port=37881
--webserver_interface=127.5.213.194
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:41:52.691000 21019 raft_consensus.cc:348] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:52.692154 21019 raft_consensus.cc:729] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efcc780097604e108f1bf64589be096a, State: Initialized, Role: FOLLOWER
I20260618 22:41:52.692816 21019 consensus_queue.cc:260] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:52.694315 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:52.693981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:46727)
0618 22:41:52.694132 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:52.694203 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260618 22:41:52.694424 21019 ts_tablet_manager.cc:1428] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260618 22:41:52.695386 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:52.695049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:46743)
0618 22:41:52.695214 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:52.695276 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260618 22:41:53.005546 21131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:53.005841 21131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:53.005934 21131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:53.013561 21131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:53.013679 21131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:53.013801 21131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:41:53.023906 21131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:32851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=37881
--tserver_master_addrs=127.5.213.254:38703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:53.025194 21131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:53.026456 21131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:53.039206 21136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:53.039248 21139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:53.039217 21137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:53.040246 21131 server_base.cc:1034] running on GCE node
I20260618 22:41:53.040864 21131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:53.041807 21131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:53.043018 21131 hybrid_clock.cc:648] HybridClock initialized: now 1781822513042967 us; error 42 us; skew 500 ppm
I20260618 22:41:53.043356 21131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:53 dist-test-slave-lnj3 krb5kdc[19814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822513, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:53.046401 21131 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:41:53.047938 21131 webserver.cc:460] Webserver started at http://127.5.213.194:37881/ using document root <none> and password file <none>
I20260618 22:41:53.048611 21131 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:53.048724 21131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:41:53.053879 21131 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:41:53.056882 21146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:53.058310 21131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:41:53.058475 21131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "eb1bc427c6104d1aa18c574cfcbeb05b"
format_stamp: "Formatted at 2026-06-18 22:41:47 on dist-test-slave-lnj3"
server_key: "351753726aae08919cd03dc30c7d0ee5"
server_key_iv: "c6e0403ac895f5689d29255b9e6cfc76"
server_key_version: "encryptionkey@0"
I20260618 22:41:53.059553 21131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:53.085258 21131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:53.089452 21131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:53.089728 21131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:53.090826 21131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:41:53.092173 21131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:41:53.092280 21131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:53.092386 21131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:41:53.092453 21131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:53.123014 21131 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:32851
I20260618 22:41:53.123085 21258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:32851 every 8 connection(s)
I20260618 22:41:53.124464 21131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:41:53.129220 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:53.127358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.194:49951)
0618 22:41:53.128078 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:53.128326 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":266,"threads_started":1}
W20260618 22:41:53.129868 21259 heartbeater.cc:644] Failed to heartbeat to 127.5.213.254:38703 (0 consecutive failures): Network error: Failed to ping master at 127.5.213.254:38703: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
I20260618 22:41:53.130716 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:53.130382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.194:44939)
0618 22:41:53.130517 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:53.130586 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260618 22:41:53.131523 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:53.131248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.194:34067)
0618 22:41:53.131356 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:53.131425 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260618 22:41:53.131842 21259 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 22:41:53.132498 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 21131
I20260618 22:41:53.133181 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:44899
--local_ip_for_outbound_sockets=127.5.213.195
--tserver_master_addrs=127.5.213.254:38703
--webserver_port=43757
--webserver_interface=127.5.213.195
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:41:53.443435 21262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:41:53.443694 21262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:41:53.443836 21262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:41:53.450947 21262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:41:53.451050 21262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:41:53.451202 21262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:41:53.460813 21262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:44899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=43757
--tserver_master_addrs=127.5.213.254:38703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:41:53.462064 21262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:41:53.463430 21262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:41:53.475605 21267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:53.475641 21270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:41:53.475711 21268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:41:53.476774 21262 server_base.cc:1034] running on GCE node
I20260618 22:41:53.477308 21262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:41:53.478178 21262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:41:53.479384 21262 hybrid_clock.cc:648] HybridClock initialized: now 1781822513479327 us; error 51 us; skew 500 ppm
I20260618 22:41:53.479676 21262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:41:53 dist-test-slave-lnj3 krb5kdc[19814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822513, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:41:53.486316 21262 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:41:53.487888 21262 webserver.cc:460] Webserver started at http://127.5.213.195:43757/ using document root <none> and password file <none>
I20260618 22:41:53.488557 21262 fs_manager.cc:359] Metadata directory not provided
I20260618 22:41:53.488672 21262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:41:53.493849 21262 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:41:53.496555 21277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:53.497941 21262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:41:53.498095 21262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "4bdb44283a794928bd76eddf3443eaa4"
format_stamp: "Formatted at 2026-06-18 22:41:48 on dist-test-slave-lnj3"
server_key: "dce57b15eb94384fc83048c5f1c6278b"
server_key_iv: "d4f2301607490b78c8aa121e25aa8dd8"
server_key_version: "encryptionkey@0"
I20260618 22:41:53.499145 21262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:41:53.526242 21262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:41:53.530681 21262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:41:53.530964 21262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:41:53.531991 21262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:41:53.533339 21262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:41:53.533432 21262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:53.533536 21262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:41:53.533603 21262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:41:53.565447 21262 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:44899
I20260618 22:41:53.565505 21389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:44899 every 8 connection(s)
I20260618 22:41:53.567049 21262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:41:53.572016 21392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:53.570084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.195:52503)
0618 22:41:53.570909 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:53.571152 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":294,"threads_started":1}
W20260618 22:41:53.572634 21390 heartbeater.cc:644] Failed to heartbeat to 127.5.213.254:38703 (0 consecutive failures): Network error: Failed to ping master at 127.5.213.254:38703: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
I20260618 22:41:53.573462 21392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:53.573149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.195:60999)
0618 22:41:53.573294 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:53.573360 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260618 22:41:53.574337 21392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:53.574021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.195:43907)
0618 22:41:53.574159 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:53.574231 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260618 22:41:53.574658 21390 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 22:41:53.574999 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 21262
I20260618 22:41:53.697096 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:53.696240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:40991)
0618 22:41:53.696739 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:53.696845 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":235,"threads_started":1}
I20260618 22:41:54.133250 21395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:54.132399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.194:45969)
0618 22:41:54.132917 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:54.133016 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":261,"threads_started":1}
I20260618 22:41:54.316912 21396 raft_consensus.cc:482] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:41:54.317175 21396 raft_consensus.cc:504] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:54.318907 21396 leader_election.cc:288] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb1bc427c6104d1aa18c574cfcbeb05b (127.5.213.194:32851), 4bdb44283a794928bd76eddf3443eaa4 (127.5.213.195:44899)
Jun 18 22:41:54 dist-test-slave-lnj3 krb5kdc[19814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822512, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.194@KRBTEST.COM
Jun 18 22:41:54 dist-test-slave-lnj3 krb5kdc[19814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822512, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.195@KRBTEST.COM
I20260618 22:41:54.339869 21398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:54.319362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:40493 (local address 127.5.213.194:32851)
0618 22:41:54.321146 (+ 1784us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:54.321158 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:54.321192 (+ 34us) server_negotiation.cc:407] Connection header received
0618 22:41:54.322993 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:54.323002 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:54.323375 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:54.323673 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:54.325653 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:54.326847 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:54.328575 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:54.329174 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:54.329342 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:54.333099 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:54.333110 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:54.333130 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:54.333178 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:54.336135 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:54.336946 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:54.336953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:54.336959 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:54.337114 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:54.337553 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:54.337557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:54.337559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:54.338073 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:54.338235 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:54.339100 (+ 865us) server_negotiation.cc:299] Negotiation successful
0618 22:41:54.339463 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1628,"thread_start_us":1515,"threads_started":1}
I20260618 22:41:54.339876 21400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:54.319781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:35201 (local address 127.5.213.195:44899)
0618 22:41:54.321454 (+ 1673us) server_negotiation.cc:206] Beginning negotiation
0618 22:41:54.321464 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:41:54.321493 (+ 29us) server_negotiation.cc:407] Connection header received
0618 22:41:54.323194 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:41:54.323202 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:41:54.323535 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:41:54.323820 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:41:54.325650 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:54.326839 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:54.328579 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:41:54.329197 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:41:54.329357 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:54.333329 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:41:54.333339 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:41:54.333356 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:41:54.333400 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:41:54.336235 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:54.337012 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:54.337018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:54.337024 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:54.337188 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:41:54.337581 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:41:54.337584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:41:54.337586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:41:54.338074 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:41:54.338238 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0618 22:41:54.339134 (+ 896us) server_negotiation.cc:299] Negotiation successful
0618 22:41:54.339456 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1529,"thread_start_us":1426,"threads_started":1}
I20260618 22:41:54.339897 21397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:54.319042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:32851 (local address 127.5.213.193:40493)
0618 22:41:54.319801 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:54.319844 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:54.320448 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:54.323827 (+ 3379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:54.323838 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:54.324710 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:54.325457 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:54.325477 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:54.327011 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:54.327015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:54.328436 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:54.328448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:54.329373 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:54.329379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:54.329660 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:54.330386 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:54.330411 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:54.332766 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:54.336290 (+ 3524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:54.336299 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:54.336314 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:54.336807 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:54.337240 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:54.337243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:54.337252 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:54.337442 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:54.338222 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:54.338230 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:54.338569 (+ 339us) client_negotiation.cc:769] Sending connection context
0618 22:41:54.338863 (+ 294us) client_negotiation.cc:240] Negotiation successful
0618 22:41:54.339425 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":234,"spinlock_wait_cycles":116992,"thread_start_us":477,"threads_started":1}
I20260618 22:41:54.340607 21399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:54.319360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:44899 (local address 127.5.213.193:35201)
0618 22:41:54.320106 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:54.320122 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:41:54.320448 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:41:54.323827 (+ 3379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:41:54.323838 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:41:54.324713 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:41:54.325457 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:54.325477 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:54.327018 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:54.327021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:54.328438 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:41:54.328449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:41:54.329398 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:41:54.329404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:41:54.329654 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:41:54.330449 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:41:54.330466 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:41:54.333059 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:41:54.336405 (+ 3346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:54.336410 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:54.336415 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:54.336877 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:54.337325 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:41:54.337331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:41:54.337334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:41:54.337474 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:41:54.338223 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:41:54.338231 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:41:54.338543 (+ 312us) client_negotiation.cc:769] Sending connection context
0618 22:41:54.338880 (+ 337us) client_negotiation.cc:240] Negotiation successful
0618 22:41:54.339568 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":392,"spinlock_wait_cycles":89344,"thread_start_us":568,"threads_started":1}
W20260618 22:41:54.343194 21013 leader_election.cc:341] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb1bc427c6104d1aa18c574cfcbeb05b (127.5.213.194:32851): Not found: Tablet not found: 081b7ff09ccf432fb6510c5c8f490518
W20260618 22:41:54.343484 21014 leader_election.cc:341] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4bdb44283a794928bd76eddf3443eaa4 (127.5.213.195:44899): Not found: Tablet not found: 081b7ff09ccf432fb6510c5c8f490518
I20260618 22:41:54.343597 21014 leader_election.cc:302] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efcc780097604e108f1bf64589be096a; no voters: 4bdb44283a794928bd76eddf3443eaa4, eb1bc427c6104d1aa18c574cfcbeb05b
I20260618 22:41:54.343931 21396 raft_consensus.cc:2726] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:41:54.576045 21401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:54.575168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.195:47835)
0618 22:41:54.575639 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:54.575801 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":219,"threads_started":1}
I20260618 22:41:54.698410 21399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:54.698016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:36827)
0618 22:41:54.698176 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:54.698272 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260618 22:41:55.135100 21402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:55.134304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.194:37529)
0618 22:41:55.134842 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:55.134942 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":276,"threads_started":1}
I20260618 22:41:55.577643 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:55.576970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.195:49863)
0618 22:41:55.577403 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:55.577488 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":225,"threads_started":1}
I20260618 22:41:55.700198 21404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:55.699377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:43295)
0618 22:41:55.699895 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:55.699983 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":227,"threads_started":1}
I20260618 22:41:55.912706 21405 raft_consensus.cc:482] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:41:55.912899 21405 raft_consensus.cc:504] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:55.913559 21405 leader_election.cc:288] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb1bc427c6104d1aa18c574cfcbeb05b (127.5.213.194:32851), 4bdb44283a794928bd76eddf3443eaa4 (127.5.213.195:44899)
W20260618 22:41:55.914578 21014 leader_election.cc:341] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4bdb44283a794928bd76eddf3443eaa4 (127.5.213.195:44899): Not found: Tablet not found: 081b7ff09ccf432fb6510c5c8f490518
W20260618 22:41:55.914798 21013 leader_election.cc:341] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb1bc427c6104d1aa18c574cfcbeb05b (127.5.213.194:32851): Not found: Tablet not found: 081b7ff09ccf432fb6510c5c8f490518
I20260618 22:41:55.914950 21013 leader_election.cc:302] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efcc780097604e108f1bf64589be096a; no voters: 4bdb44283a794928bd76eddf3443eaa4, eb1bc427c6104d1aa18c574cfcbeb05b
I20260618 22:41:55.915197 21405 raft_consensus.cc:2726] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:41:56.136835 21406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:56.136058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.194:54223)
0618 22:41:56.136529 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:56.136624 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":255,"threads_started":1}
I20260618 22:41:56.579401 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:56.578606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.195:34563)
0618 22:41:56.579144 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:56.579239 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":259,"threads_started":1}
I20260618 22:41:56.701875 21408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:56.701138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:43717)
0618 22:41:56.701584 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:56.701710 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":224,"threads_started":1}
I20260618 22:41:57.138486 21409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:57.137731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.194:52323)
0618 22:41:57.138216 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:57.138302 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":225,"threads_started":1}
I20260618 22:41:57.581193 21410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:57.580404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.195:48709)
0618 22:41:57.580917 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:57.581011 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":264,"threads_started":1}
I20260618 22:41:57.703671 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:57.702969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:37509)
0618 22:41:57.703407 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:57.703490 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":212,"threads_started":1}
I20260618 22:41:58.140151 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:58.139414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.194:54325)
0618 22:41:58.139883 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:58.139971 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":262,"threads_started":1}
I20260618 22:41:58.583112 21413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:58.582267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.195:34637)
0618 22:41:58.582825 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:58.582918 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":255,"threads_started":1}
I20260618 22:41:58.705354 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:58.704613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:37329)
0618 22:41:58.705088 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:58.705178 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":258,"threads_started":1}
I20260618 22:41:59.132607 21415 raft_consensus.cc:482] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:41:59.132910 21415 raft_consensus.cc:504] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:41:59.133608 21415 leader_election.cc:288] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb1bc427c6104d1aa18c574cfcbeb05b (127.5.213.194:32851), 4bdb44283a794928bd76eddf3443eaa4 (127.5.213.195:44899)
W20260618 22:41:59.134549 21014 leader_election.cc:341] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4bdb44283a794928bd76eddf3443eaa4 (127.5.213.195:44899): Not found: Tablet not found: 081b7ff09ccf432fb6510c5c8f490518
W20260618 22:41:59.134724 21013 leader_election.cc:341] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb1bc427c6104d1aa18c574cfcbeb05b (127.5.213.194:32851): Not found: Tablet not found: 081b7ff09ccf432fb6510c5c8f490518
I20260618 22:41:59.134868 21013 leader_election.cc:302] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efcc780097604e108f1bf64589be096a; no voters: 4bdb44283a794928bd76eddf3443eaa4, eb1bc427c6104d1aa18c574cfcbeb05b
I20260618 22:41:59.135155 21415 raft_consensus.cc:2726] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:41:59.142019 21416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:59.141117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.194:41285)
0618 22:41:59.141697 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:59.141835 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":344,"threads_started":1}
I20260618 22:41:59.584976 21417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:59.584204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.195:49757)
0618 22:41:59.584708 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:59.584802 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":249,"threads_started":1}
I20260618 22:41:59.707120 21418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:41:59.706303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:33527)
0618 22:41:59.706812 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0618 22:41:59.706904 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":247,"threads_started":1}
I20260618 22:42:00.143858 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:00.143007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.194:46995)
0618 22:42:00.143571 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:00.143669 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":256,"threads_started":1}
I20260618 22:42:00.586896 21420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:00.585969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.195:34099)
0618 22:42:00.586500 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:00.586681 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":242,"threads_started":1}
I20260618 22:42:00.708740 21421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:00.708034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:57835)
0618 22:42:00.708488 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:00.708569 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":220,"threads_started":1}
I20260618 22:42:00.750490 21422 raft_consensus.cc:482] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:42:00.750744 21422 raft_consensus.cc:504] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:42:00.751363 21422 leader_election.cc:288] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb1bc427c6104d1aa18c574cfcbeb05b (127.5.213.194:32851), 4bdb44283a794928bd76eddf3443eaa4 (127.5.213.195:44899)
W20260618 22:42:00.752362 21014 leader_election.cc:341] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4bdb44283a794928bd76eddf3443eaa4 (127.5.213.195:44899): Not found: Tablet not found: 081b7ff09ccf432fb6510c5c8f490518
W20260618 22:42:00.752548 21013 leader_election.cc:341] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb1bc427c6104d1aa18c574cfcbeb05b (127.5.213.194:32851): Not found: Tablet not found: 081b7ff09ccf432fb6510c5c8f490518
I20260618 22:42:00.752732 21013 leader_election.cc:302] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efcc780097604e108f1bf64589be096a; no voters: 4bdb44283a794928bd76eddf3443eaa4, eb1bc427c6104d1aa18c574cfcbeb05b
I20260618 22:42:00.752957 21422 raft_consensus.cc:2726] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:42:01.145627 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:01.144878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.194:52069)
0618 22:42:01.145364 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:01.145450 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":249,"threads_started":1}
I20260618 22:42:01.588663 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:01.587979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.195:57859)
0618 22:42:01.588401 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:01.588487 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":215,"threads_started":1}
I20260618 22:42:01.710414 21425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:01.709667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:41429)
0618 22:42:01.710144 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:01.710239 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":239,"threads_started":1}
I20260618 22:42:02.147603 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:02.146704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.194:44667)
0618 22:42:02.147209 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:02.147435 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":216,"threads_started":1}
I20260618 22:42:02.590373 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:02.589592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.195:48781)
0618 22:42:02.590118 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:02.590211 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":271,"threads_started":1}
I20260618 22:42:02.712134 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:02.711426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:35181)
0618 22:42:02.711889 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:02.711969 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":252,"threads_started":1}
I20260618 22:42:03.149413 21429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:03.148710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.194:48009)
0618 22:42:03.149150 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:03.149238 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":215,"threads_started":1}
I20260618 22:42:03.592072 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:03.591294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.195:40951)
0618 22:42:03.591785 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:03.591888 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":264,"threads_started":1}
I20260618 22:42:03.713805 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:03.713056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:39589)
0618 22:42:03.713486 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:03.713581 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":218,"threads_started":1}
I20260618 22:42:04.151481 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:04.150439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.194:49495)
0618 22:42:04.151148 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:04.151269 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":363,"threads_started":1}
I20260618 22:42:04.593863 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:04.593014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.195:53585)
0618 22:42:04.593539 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:04.593658 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":274,"threads_started":1}
I20260618 22:42:04.715550 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:04.714796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:35239)
0618 22:42:04.715258 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:04.715353 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":230,"threads_started":1}
I20260618 22:42:05.153201 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:05.152488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.194:36389)
0618 22:42:05.152949 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:05.153031 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":249,"threads_started":1}
I20260618 22:42:05.595630 21436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:05.594934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.195:45275)
0618 22:42:05.595379 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:05.595465 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":226,"threads_started":1}
I20260618 22:42:05.717267 21437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:05.716537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:40127)
0618 22:42:05.717015 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:05.717096 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":266,"threads_started":1}
I20260618 22:42:06.154959 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:06.154154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.194:36953)
0618 22:42:06.154671 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:06.154801 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":246,"threads_started":1}
I20260618 22:42:06.526397 21439 raft_consensus.cc:482] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:42:06.526572 21439 raft_consensus.cc:504] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb1bc427c6104d1aa18c574cfcbeb05b" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32851 } } peers { permanent_uuid: "efcc780097604e108f1bf64589be096a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44055 } } peers { permanent_uuid: "4bdb44283a794928bd76eddf3443eaa4" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44899 } }
I20260618 22:42:06.527256 21439 leader_election.cc:288] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb1bc427c6104d1aa18c574cfcbeb05b (127.5.213.194:32851), 4bdb44283a794928bd76eddf3443eaa4 (127.5.213.195:44899)
W20260618 22:42:06.528153 21013 leader_election.cc:341] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb1bc427c6104d1aa18c574cfcbeb05b (127.5.213.194:32851): Not found: Tablet not found: 081b7ff09ccf432fb6510c5c8f490518
W20260618 22:42:06.528317 21014 leader_election.cc:341] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4bdb44283a794928bd76eddf3443eaa4 (127.5.213.195:44899): Not found: Tablet not found: 081b7ff09ccf432fb6510c5c8f490518
I20260618 22:42:06.528456 21014 leader_election.cc:302] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efcc780097604e108f1bf64589be096a; no voters: 4bdb44283a794928bd76eddf3443eaa4, eb1bc427c6104d1aa18c574cfcbeb05b
I20260618 22:42:06.528687 21439 raft_consensus.cc:2726] T 081b7ff09ccf432fb6510c5c8f490518 P efcc780097604e108f1bf64589be096a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:42:06.597440 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:06.596654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.195:49313)
0618 22:42:06.597114 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:06.597214 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":231,"threads_started":1}
I20260618 22:42:06.719126 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:06.718232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:44759)
0618 22:42:06.718781 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:06.718896 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":257,"threads_started":1}
I20260618 22:42:07.156665 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:07.155915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.194:53065)
0618 22:42:07.156383 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:07.156471 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":216,"threads_started":1}
I20260618 22:42:07.599107 21443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:07.598320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.195:38757)
0618 22:42:07.598821 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:07.598902 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":232,"threads_started":1}
I20260618 22:42:07.721216 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:07.720447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.193:34205)
0618 22:42:07.720954 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:07.721043 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":228,"threads_started":1}
I20260618 22:42:08.158806 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:08.157855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.194:56237)
0618 22:42:08.158487 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:08.158592 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":360,"threads_started":1}
I20260618 22:42:08.579896 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 20963
I20260618 22:42:08.595939 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:08.581092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48652 (local address 127.5.213.193:44055)
0618 22:42:08.581844 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:08.581851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:08.581884 (+ 33us) server_negotiation.cc:407] Connection header received
0618 22:42:08.581975 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:08.581985 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:08.582276 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:08.582515 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:08.584440 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:08.585462 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:08.586842 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:08.587466 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:08.587587 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:08.590258 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:08.590272 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:08.590288 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:08.590328 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:08.593198 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:08.593930 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:08.593937 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:08.593939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:08.594083 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:08.594460 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:08.594464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:08.594466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:08.594896 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:08.595157 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:08.595416 (+ 259us) server_negotiation.cc:299] Negotiation successful
0618 22:42:08.595617 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":457,"threads_started":1}
I20260618 22:42:08.597183 21122 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:42:08.601030 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:08.600249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:38703 (local address 127.5.213.195:55393)
0618 22:42:08.600770 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:08.600881 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:38703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":239,"threads_started":1}
I20260618 22:42:08.686120 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 20963
I20260618 22:42:08.701094 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 21131
I20260618 22:42:08.715019 21449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:08.701964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59000 (local address 127.5.213.194:32851)
0618 22:42:08.702437 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:08.702442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:08.702458 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:42:08.702510 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:08.702514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:08.702573 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:08.702700 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:08.704020 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:08.704803 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:08.706179 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:08.706612 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:08.706761 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:08.709474 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:08.709479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:08.709482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:08.709519 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:08.712215 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:08.712975 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:08.712979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:08.712982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:08.713116 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:08.713557 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:08.713562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:08.713564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:08.713869 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:08.714079 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:08.714435 (+ 356us) server_negotiation.cc:299] Negotiation successful
0618 22:42:08.714621 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":259,"threads_started":1}
I20260618 22:42:08.715792 21254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:42:08.799073 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 21131
I20260618 22:42:08.814993 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 21262
I20260618 22:42:08.828635 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:08.815820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59404 (local address 127.5.213.195:44899)
0618 22:42:08.816375 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:08.816380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:08.816397 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:42:08.816451 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:08.816454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:08.816519 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:08.816615 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:08.818005 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:08.818828 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:08.820081 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:08.820515 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:08.820627 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:08.823224 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:08.823229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:08.823232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:08.823268 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:08.825852 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:08.826568 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:08.826574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:08.826578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:08.826802 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:08.827154 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:08.827159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:08.827163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:08.827452 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:08.827697 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:08.828031 (+ 334us) server_negotiation.cc:299] Negotiation successful
0618 22:42:08.828248 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":267,"threads_started":1}
I20260618 22:42:08.829365 21385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:42:08.912326 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 21262
2026-06-18T22:42:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23989 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:42:09 dist-test-slave-lnj3 krb5kdc[21455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:42:09 dist-test-slave-lnj3 krb5kdc[21455](info): set up 2 sockets
Jun 18 22:42:09 dist-test-slave-lnj3 krb5kdc[21455](info): commencing operation
krb5kdc: starting...
W20260618 22:42:09.919062 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:42:10 dist-test-slave-lnj3 krb5kdc[21455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:42:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:42:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:42:10.143725 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:45069
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:45069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:42:10.464666 21468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:42:10.464985 21468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:42:10.465121 21468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:42:10.472410 21468 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260618 22:42:10.472518 21468 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260618 22:42:10.472573 21468 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260618 22:42:10.472648 21468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:42:10.472721 21468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:42:10.472815 21468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:42:10.472874 21468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:42:10.472947 21468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:42:10.483155 21468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:45069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:45069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:42:10.484455 21468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:42:10.485770 21468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:42:10.497483 21473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:10.497551 21474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:10.497524 21476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:42:10.498175 21468 server_base.cc:1034] running on GCE node
I20260618 22:42:10.499007 21468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:42:10.500435 21468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:42:10.501713 21468 hybrid_clock.cc:648] HybridClock initialized: now 1781822530501617 us; error 39 us; skew 500 ppm
I20260618 22:42:10.502074 21468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:42:10 dist-test-slave-lnj3 krb5kdc[21455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822530, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:42:10.505267 21468 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:42:10.506922 21468 webserver.cc:460] Webserver started at http://127.5.213.254:36885/ using document root <none> and password file <none>
I20260618 22:42:10.507632 21468 fs_manager.cc:359] Metadata directory not provided
I20260618 22:42:10.507735 21468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:42:10.508029 21468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:42:10.511071 21468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "9c06746ac2c24833879dde032c0ac934"
format_stamp: "Formatted at 2026-06-18 22:42:10 on dist-test-slave-lnj3"
server_key: "4f0ec6b4daa8cfce989761822a24395e"
server_key_iv: "d6dbb391f4db6184aee08c9ccf6fbc3e"
server_key_version: "encryptionkey@0"
I20260618 22:42:10.511833 21468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "9c06746ac2c24833879dde032c0ac934"
format_stamp: "Formatted at 2026-06-18 22:42:10 on dist-test-slave-lnj3"
server_key: "4f0ec6b4daa8cfce989761822a24395e"
server_key_iv: "d6dbb391f4db6184aee08c9ccf6fbc3e"
server_key_version: "encryptionkey@0"
I20260618 22:42:10.516995 21468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260618 22:42:10.520457 21483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:10.521916 21468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260618 22:42:10.522090 21468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "9c06746ac2c24833879dde032c0ac934"
format_stamp: "Formatted at 2026-06-18 22:42:10 on dist-test-slave-lnj3"
server_key: "4f0ec6b4daa8cfce989761822a24395e"
server_key_iv: "d6dbb391f4db6184aee08c9ccf6fbc3e"
server_key_version: "encryptionkey@0"
I20260618 22:42:10.522300 21468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:42:10.546990 21468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:42:10.551366 21468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:42:10.551631 21468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:42:10.569233 21468 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:45069
I20260618 22:42:10.569259 21534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:45069 every 8 connection(s)
I20260618 22:42:10.570598 21468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:42:10.574421 21535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:42:10.575416 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 21468
I20260618 22:42:10.575610 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:42:10.576241 5975 external_mini_cluster.cc:1428] Setting key 6524ec9ef082e5e4b2bd4ba8000e1374
I20260618 22:42:10.587036 21535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934: Bootstrap starting.
Jun 18 22:42:10 dist-test-slave-lnj3 krb5kdc[21455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:42:10.590265 21535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934: Neither blocks nor log segments found. Creating new log.
I20260618 22:42:10.591566 21535 log.cc:824] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934: Log is configured to *not* fsync() on all Append() calls
I20260618 22:42:10.595163 21535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934: No bootstrap required, opened a new log
I20260618 22:42:10.599702 21538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:10.578341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.5.213.254:45069)
0618 22:42:10.579156 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:10.579168 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:10.579212 (+ 44us) server_negotiation.cc:407] Connection header received
0618 22:42:10.580782 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:10.580808 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:10.581207 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:10.581587 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:10.583076 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:10.584375 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:10.585782 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:10.586449 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:10.586620 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:10.590853 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:10.590869 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:10.590888 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:10.590934 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:10.594847 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:10.595572 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:10.595582 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:10.595593 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:10.595806 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:10.596202 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:10.596207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:10.596211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:10.596908 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:10.597362 (+ 454us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:10.597739 (+ 377us) server_negotiation.cc:299] Negotiation successful
0618 22:42:10.598244 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":375,"threads_started":1}
I20260618 22:42:10.604622 21535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c06746ac2c24833879dde032c0ac934" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 45069 } }
I20260618 22:42:10.604976 21535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:42:10.605051 21535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c06746ac2c24833879dde032c0ac934, State: Initialized, Role: FOLLOWER
I20260618 22:42:10.605643 21535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c06746ac2c24833879dde032c0ac934" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 45069 } }
I20260618 22:42:10.605865 21535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:42:10.606021 21535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:42:10.606161 21535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:42:10.608377 21535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c06746ac2c24833879dde032c0ac934" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 45069 } }
I20260618 22:42:10.608850 21535 leader_election.cc:302] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c06746ac2c24833879dde032c0ac934; no voters:
I20260618 22:42:10.609385 21535 leader_election.cc:288] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:42:10.609512 21540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:42:10.610100 21540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 1 LEADER]: Becoming Leader. State: Replica: 9c06746ac2c24833879dde032c0ac934, State: Running, Role: LEADER
I20260618 22:42:10.610683 21540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c06746ac2c24833879dde032c0ac934" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 45069 } }
I20260618 22:42:10.611377 21535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:42:10.613247 21542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c06746ac2c24833879dde032c0ac934. Latest consensus state: current_term: 1 leader_uuid: "9c06746ac2c24833879dde032c0ac934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c06746ac2c24833879dde032c0ac934" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 45069 } } }
I20260618 22:42:10.613744 21542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [sys.catalog]: This master's current role is: LEADER
I20260618 22:42:10.613724 21541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c06746ac2c24833879dde032c0ac934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c06746ac2c24833879dde032c0ac934" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 45069 } } }
I20260618 22:42:10.614555 21541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [sys.catalog]: This master's current role is: LEADER
I20260618 22:42:10.615780 21545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:42:10.621030 21545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:42:10.629911 21545 catalog_manager.cc:1347] Generated new cluster ID: 7b998e51a5fb427491a1a0ade82e0a8c
I20260618 22:42:10.630016 21545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:42:10.649338 21545 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:42:10.650731 21545 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:42:10.662957 21545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934: Generated new TSK 0
I20260618 22:42:10.663964 21545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
I20260618 22:42:10.729903 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:45069
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:42:11.053942 21561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:42:11.054255 21561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:42:11.054376 21561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:42:11.061980 21561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:42:11.062108 21561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:42:11.062264 21561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:42:11.072355 21561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:45069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:42:11.073807 21561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:42:11.075214 21561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:42:11.087868 21566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:11.087980 21567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:11.088335 21569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:42:11.089035 21561 server_base.cc:1034] running on GCE node
I20260618 22:42:11.089625 21561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:42:11.090659 21561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:42:11.091876 21561 hybrid_clock.cc:648] HybridClock initialized: now 1781822531091820 us; error 52 us; skew 500 ppm
I20260618 22:42:11.092190 21561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:42:11 dist-test-slave-lnj3 krb5kdc[21455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822531, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:42:11.095363 21561 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:42:11.096829 21561 webserver.cc:460] Webserver started at http://127.5.213.193:37697/ using document root <none> and password file <none>
I20260618 22:42:11.097486 21561 fs_manager.cc:359] Metadata directory not provided
I20260618 22:42:11.097582 21561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:42:11.097857 21561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:42:11.100868 21561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "573d81b018874aa3a49e60a70914643a"
format_stamp: "Formatted at 2026-06-18 22:42:11 on dist-test-slave-lnj3"
server_key: "d970bca08826606cebdc5be6c8c3d04e"
server_key_iv: "77c5d1e46d0f92258e064582dd2fc8ca"
server_key_version: "encryptionkey@0"
I20260618 22:42:11.101707 21561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "573d81b018874aa3a49e60a70914643a"
format_stamp: "Formatted at 2026-06-18 22:42:11 on dist-test-slave-lnj3"
server_key: "d970bca08826606cebdc5be6c8c3d04e"
server_key_iv: "77c5d1e46d0f92258e064582dd2fc8ca"
server_key_version: "encryptionkey@0"
I20260618 22:42:11.107092 21561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260618 22:42:11.110797 21576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:11.112360 21561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:42:11.112527 21561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "573d81b018874aa3a49e60a70914643a"
format_stamp: "Formatted at 2026-06-18 22:42:11 on dist-test-slave-lnj3"
server_key: "d970bca08826606cebdc5be6c8c3d04e"
server_key_iv: "77c5d1e46d0f92258e064582dd2fc8ca"
server_key_version: "encryptionkey@0"
I20260618 22:42:11.112751 21561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:42:11.129451 21561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:42:11.133793 21561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:42:11.134073 21561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:42:11.135068 21561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:42:11.136445 21561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:42:11.136533 21561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:11.136664 21561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:42:11.136725 21561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:11.166249 21561 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:35929
I20260618 22:42:11.166314 21688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:35929 every 8 connection(s)
I20260618 22:42:11.167632 21561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:42:11.171849 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 21561
I20260618 22:42:11.172029 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:42:11.172626 5975 external_mini_cluster.cc:1428] Setting key f35a968aa20c4a46c1f671cce2e9fa64
Jun 18 22:42:11 dist-test-slave-lnj3 krb5kdc[21455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822531, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:42:11.189505 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:11.170074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:45719 (local address 127.5.213.254:45069)
0618 22:42:11.170500 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:11.170504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:11.171451 (+ 947us) server_negotiation.cc:407] Connection header received
0618 22:42:11.173556 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:11.173560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:11.173640 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:11.173743 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:11.175775 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:11.176529 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:11.178712 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:11.179118 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:11.179227 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:11.183209 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:11.183215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:11.183220 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:11.183265 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:11.186295 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:11.187045 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:11.187050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:11.187053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:11.187191 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:11.187623 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:11.187627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:11.187630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:11.187981 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:11.188121 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:11.188937 (+ 816us) server_negotiation.cc:299] Negotiation successful
0618 22:42:11.189169 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":218,"threads_started":1}
I20260618 22:42:11.190418 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:11.170539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.193:45719)
0618 22:42:11.171294 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:11.171333 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:42:11.173145 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:42:11.174023 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:42:11.174033 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:42:11.174751 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:42:11.175578 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:11.175603 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:11.176693 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:11.176697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:11.178552 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:11.178567 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:11.179351 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:11.179358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:11.179653 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:11.180366 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:42:11.180394 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:42:11.182962 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:42:11.186433 (+ 3471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:42:11.186446 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:42:11.186460 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:42:11.186930 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:42:11.187321 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:42:11.187325 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:42:11.187327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:42:11.187522 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:42:11.188124 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:42:11.188132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:42:11.188496 (+ 364us) client_negotiation.cc:769] Sending connection context
0618 22:42:11.188798 (+ 302us) client_negotiation.cc:240] Negotiation successful
0618 22:42:11.189218 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":265,"threads_started":1}
I20260618 22:42:11.192126 21689 heartbeater.cc:347] Connected to a master server at 127.5.213.254:45069
I20260618 22:42:11.192454 21689 heartbeater.cc:464] Registering TS with master...
I20260618 22:42:11.193390 21689 heartbeater.cc:511] Master 127.5.213.254:45069 requested a full tablet report, sending...
I20260618 22:42:11.196247 21500 ts_manager.cc:194] Registered new tserver with Master: 573d81b018874aa3a49e60a70914643a (127.5.213.193:35929)
WARNING: no policy specified for kudu/127.5.213.194@KRBTEST.COM; defaulting to no policy
I20260618 22:42:11.197758 21500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:45719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
I20260618 22:42:11.223973 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:45069
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:42:11.538484 21695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:42:11.538843 21695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:42:11.538992 21695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:42:11.546277 21695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:42:11.546401 21695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:42:11.546558 21695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:42:11.556522 21695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:45069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:42:11.557818 21695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:42:11.559118 21695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:42:11.572190 21701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:11.572546 21700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:11.572767 21703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:42:11.573069 21695 server_base.cc:1034] running on GCE node
I20260618 22:42:11.573818 21695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:42:11.575033 21695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:42:11.576261 21695 hybrid_clock.cc:648] HybridClock initialized: now 1781822531576208 us; error 47 us; skew 500 ppm
I20260618 22:42:11.576656 21695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:42:11 dist-test-slave-lnj3 krb5kdc[21455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822531, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:42:11.580613 21695 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:42:11.582226 21695 webserver.cc:460] Webserver started at http://127.5.213.194:38561/ using document root <none> and password file <none>
I20260618 22:42:11.582953 21695 fs_manager.cc:359] Metadata directory not provided
I20260618 22:42:11.583076 21695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:42:11.583370 21695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:42:11.586346 21695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "09b731a378054e73831d2313a823a2df"
format_stamp: "Formatted at 2026-06-18 22:42:11 on dist-test-slave-lnj3"
server_key: "ac955d938b10fc03cf0ad3f4c4b84024"
server_key_iv: "6a497b162b618903fef4ea74e81a09fe"
server_key_version: "encryptionkey@0"
I20260618 22:42:11.587146 21695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "09b731a378054e73831d2313a823a2df"
format_stamp: "Formatted at 2026-06-18 22:42:11 on dist-test-slave-lnj3"
server_key: "ac955d938b10fc03cf0ad3f4c4b84024"
server_key_iv: "6a497b162b618903fef4ea74e81a09fe"
server_key_version: "encryptionkey@0"
I20260618 22:42:11.592381 21695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260618 22:42:11.596042 21710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:11.597476 21695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:42:11.597631 21695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "09b731a378054e73831d2313a823a2df"
format_stamp: "Formatted at 2026-06-18 22:42:11 on dist-test-slave-lnj3"
server_key: "ac955d938b10fc03cf0ad3f4c4b84024"
server_key_iv: "6a497b162b618903fef4ea74e81a09fe"
server_key_version: "encryptionkey@0"
I20260618 22:42:11.597816 21695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:42:11.627933 21695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:42:11.632112 21695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:42:11.632388 21695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:42:11.633309 21695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:42:11.634567 21695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:42:11.634665 21695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:11.634793 21695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:42:11.634858 21695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:11.664644 21695 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:40135
I20260618 22:42:11.664695 21822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:40135 every 8 connection(s)
I20260618 22:42:11.665921 21695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:42:11.675588 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 21695
I20260618 22:42:11.675804 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:42:11.676348 5975 external_mini_cluster.cc:1428] Setting key 86bf77b9a13ad629e520f9deee926a0e
Jun 18 22:42:11 dist-test-slave-lnj3 krb5kdc[21455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822531, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:42:11.686770 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:11.668425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:57851 (local address 127.5.213.254:45069)
0618 22:42:11.668638 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:11.668644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:11.669893 (+ 1249us) server_negotiation.cc:407] Connection header received
0618 22:42:11.671888 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:11.671895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:11.671977 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:11.672063 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:11.673610 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:11.674350 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:11.676058 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:11.676722 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:11.676859 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:11.680641 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:11.680647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:11.680652 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:11.680690 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:11.683752 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:11.684516 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:11.684521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:11.684524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:11.684672 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:11.685048 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:11.685052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:11.685055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:11.685393 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:11.685539 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:11.686296 (+ 757us) server_negotiation.cc:299] Negotiation successful
0618 22:42:11.686482 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260618 22:42:11.688189 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:11.668971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.194:57851)
0618 22:42:11.669697 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:11.669747 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0618 22:42:11.671611 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:42:11.672329 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:42:11.672338 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:42:11.672861 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:42:11.673439 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:11.673458 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:11.674498 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:11.674502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:11.675855 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:11.675869 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:11.676943 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:11.676949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:11.677269 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:11.677890 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:42:11.677913 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:42:11.680394 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:42:11.683939 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:42:11.683949 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:42:11.683965 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:42:11.684394 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:42:11.684763 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:42:11.684766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:42:11.684769 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:42:11.684953 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:42:11.685539 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:42:11.685546 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:42:11.685922 (+ 376us) client_negotiation.cc:769] Sending connection context
0618 22:42:11.686189 (+ 267us) client_negotiation.cc:240] Negotiation successful
0618 22:42:11.686801 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":284,"threads_started":1}
I20260618 22:42:11.689670 21823 heartbeater.cc:347] Connected to a master server at 127.5.213.254:45069
I20260618 22:42:11.689977 21823 heartbeater.cc:464] Registering TS with master...
I20260618 22:42:11.690742 21823 heartbeater.cc:511] Master 127.5.213.254:45069 requested a full tablet report, sending...
I20260618 22:42:11.692237 21500 ts_manager.cc:194] Registered new tserver with Master: 09b731a378054e73831d2313a823a2df (127.5.213.194:40135)
I20260618 22:42:11.693041 21500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:57851
WARNING: no policy specified for kudu/127.5.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
I20260618 22:42:11.726190 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:45069
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:42:12.036401 21828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:42:12.036691 21828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:42:12.036811 21828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:42:12.043933 21828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:42:12.044037 21828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:42:12.044183 21828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:42:12.053783 21828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:45069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:42:12.055028 21828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:42:12.056268 21828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:42:12.068563 21834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:12.068565 21833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:12.068804 21836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:42:12.069286 21828 server_base.cc:1034] running on GCE node
I20260618 22:42:12.069886 21828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:42:12.070798 21828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:42:12.072009 21828 hybrid_clock.cc:648] HybridClock initialized: now 1781822532071950 us; error 45 us; skew 500 ppm
I20260618 22:42:12.072336 21828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:42:12 dist-test-slave-lnj3 krb5kdc[21455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822532, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:42:12.078586 21828 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:42:12.080026 21828 webserver.cc:460] Webserver started at http://127.5.213.195:43319/ using document root <none> and password file <none>
I20260618 22:42:12.080678 21828 fs_manager.cc:359] Metadata directory not provided
I20260618 22:42:12.080780 21828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:42:12.081033 21828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:42:12.083897 21828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "20848997b00049b8a471edcf3a537b39"
format_stamp: "Formatted at 2026-06-18 22:42:12 on dist-test-slave-lnj3"
server_key: "553c035740e5989fe68d908038d4840c"
server_key_iv: "73d7b5f1aec13a420a56553bc2c84df6"
server_key_version: "encryptionkey@0"
I20260618 22:42:12.084621 21828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "20848997b00049b8a471edcf3a537b39"
format_stamp: "Formatted at 2026-06-18 22:42:12 on dist-test-slave-lnj3"
server_key: "553c035740e5989fe68d908038d4840c"
server_key_iv: "73d7b5f1aec13a420a56553bc2c84df6"
server_key_version: "encryptionkey@0"
I20260618 22:42:12.089612 21828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260618 22:42:12.092859 21843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:12.094128 21828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:42:12.094298 21828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "20848997b00049b8a471edcf3a537b39"
format_stamp: "Formatted at 2026-06-18 22:42:12 on dist-test-slave-lnj3"
server_key: "553c035740e5989fe68d908038d4840c"
server_key_iv: "73d7b5f1aec13a420a56553bc2c84df6"
server_key_version: "encryptionkey@0"
I20260618 22:42:12.094485 21828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:42:12.116813 21828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:42:12.120847 21828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:42:12.121104 21828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:42:12.122057 21828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:42:12.123425 21828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:42:12.123503 21828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:12.123615 21828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:42:12.123667 21828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:12.153235 21828 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:34669
I20260618 22:42:12.153301 21955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:34669 every 8 connection(s)
I20260618 22:42:12.154500 21828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:42:12.157275 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 21828
I20260618 22:42:12.157459 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:42:12.157930 5975 external_mini_cluster.cc:1428] Setting key 7f16297d6acfb2b5cca7baaa12feae26
Jun 18 22:42:12 dist-test-slave-lnj3 krb5kdc[21455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822532, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:42:12.174774 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:12.157069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:47101 (local address 127.5.213.254:45069)
0618 22:42:12.157281 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:12.157287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:12.158524 (+ 1237us) server_negotiation.cc:407] Connection header received
0618 22:42:12.160494 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:12.160499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:12.160581 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:12.160662 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:12.162115 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.162917 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.164399 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.164853 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.164962 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:12.168495 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:12.168502 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:12.168508 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:12.168546 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:12.171800 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:12.172560 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:12.172566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:12.172570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:12.172718 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:12.173102 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:12.173108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:12.173111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:12.173386 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:12.173519 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:12.174192 (+ 673us) server_negotiation.cc:299] Negotiation successful
0618 22:42:12.174411 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260618 22:42:12.175695 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:12.157565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.195:47101)
0618 22:42:12.158352 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:12.158402 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0618 22:42:12.160244 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:42:12.160894 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:42:12.160901 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:42:12.161407 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:42:12.161958 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:12.161979 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.163060 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.163064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:12.164268 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:12.164280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.165062 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.165068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:12.165290 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:12.165991 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:42:12.166015 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:42:12.168274 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:42:12.171916 (+ 3642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:42:12.171923 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:42:12.171934 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:42:12.172455 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:42:12.172822 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:42:12.172826 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:42:12.172829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:42:12.173002 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:42:12.173510 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:42:12.173518 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:42:12.173844 (+ 326us) client_negotiation.cc:769] Sending connection context
0618 22:42:12.174082 (+ 238us) client_negotiation.cc:240] Negotiation successful
0618 22:42:12.174591 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":307,"threads_started":1}
I20260618 22:42:12.177073 21956 heartbeater.cc:347] Connected to a master server at 127.5.213.254:45069
I20260618 22:42:12.177407 21956 heartbeater.cc:464] Registering TS with master...
I20260618 22:42:12.178162 21956 heartbeater.cc:511] Master 127.5.213.254:45069 requested a full tablet report, sending...
I20260618 22:42:12.179672 21500 ts_manager.cc:194] Registered new tserver with Master: 20848997b00049b8a471edcf3a537b39 (127.5.213.195:34669)
I20260618 22:42:12.180519 21500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:47101
I20260618 22:42:12.185271 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:42:12.200941 21689 heartbeater.cc:503] Master 127.5.213.254:45069 was elected leader, sending a full tablet report...
I20260618 22:42:12.203472 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:12.189707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48670 (local address 127.5.213.254:45069)
0618 22:42:12.189917 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:12.189922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:12.190294 (+ 372us) server_negotiation.cc:407] Connection header received
0618 22:42:12.190431 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:12.190435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:12.190500 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:12.190587 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:12.191974 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.192753 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.194120 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.194546 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.194681 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:12.197355 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:12.197363 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:12.197369 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:12.197413 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:12.200571 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:12.201327 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:12.201333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:12.201337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:12.201490 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:12.201917 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:12.201926 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:12.201928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:12.202346 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:12.202574 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:12.203015 (+ 441us) server_negotiation.cc:299] Negotiation successful
0618 22:42:12.203193 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260618 22:42:12.207288 21500 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:42:12.214231 21500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:42:12.249868 21970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:12.227341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:40135 (local address 127.0.0.1:35050)
0618 22:42:12.239286 (+ 11945us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:12.239350 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0618 22:42:12.239603 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:42:12.241610 (+ 2007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:42:12.241618 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:42:12.241649 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:42:12.241930 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:12.241939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.243514 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.243519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:12.245647 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:12.245663 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.248656 (+ 2993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.248663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:12.248914 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:12.248966 (+ 52us) client_negotiation.cc:769] Sending connection context
0618 22:42:12.249107 (+ 141us) client_negotiation.cc:240] Negotiation successful
0618 22:42:12.249340 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11575,"mutex_wait_us":1,"thread_start_us":13479,"threads_started":1}
I20260618 22:42:12.250178 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:12.228011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35929 (local address 127.0.0.1:45906)
0618 22:42:12.241140 (+ 13129us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:12.241160 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:42:12.241314 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:42:12.242050 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:42:12.242055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:42:12.242081 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:42:12.242427 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:12.242438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.244517 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.244523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:12.246930 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:12.246948 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.249343 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.249348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:12.249476 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:12.249502 (+ 26us) client_negotiation.cc:769] Sending connection context
0618 22:42:12.249578 (+ 76us) client_negotiation.cc:240] Negotiation successful
0618 22:42:12.249714 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12994,"thread_start_us":12375,"threads_started":1}
I20260618 22:42:12.250164 21971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:12.228418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.5.213.193:35929)
0618 22:42:12.240354 (+ 11936us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:12.240364 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:12.241446 (+ 1082us) server_negotiation.cc:407] Connection header received
0618 22:42:12.241554 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:12.241561 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:12.241830 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:12.242046 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:42:12.242615 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.244332 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.247157 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.248823 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.248965 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:12.249201 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:12.249657 (+ 456us) server_negotiation.cc:299] Negotiation successful
0618 22:42:12.249841 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11768,"thread_start_us":11617,"threads_started":1}
I20260618 22:42:12.251348 21969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:12.227560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35050 (local address 127.5.213.194:40135)
0618 22:42:12.240662 (+ 13102us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:12.240670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:12.240701 (+ 31us) server_negotiation.cc:407] Connection header received
0618 22:42:12.240824 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:12.240832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:12.241108 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:12.241336 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:42:12.242104 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.243361 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.245833 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.248089 (+ 2256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.248230 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:12.248475 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:12.250852 (+ 2377us) server_negotiation.cc:299] Negotiation successful
0618 22:42:12.251038 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12956,"thread_start_us":12858,"threads_started":1}
I20260618 22:42:12.255120 21624 tablet_service.cc:1467] Processing CreateTablet for tablet e08ff99b70644764aa98fa3f0a9e8906 (DEFAULT_TABLE table=test-table [id=7599d846f23a4be7b3d16137405031b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:42:12.255313 21758 tablet_service.cc:1467] Processing CreateTablet for tablet e08ff99b70644764aa98fa3f0a9e8906 (DEFAULT_TABLE table=test-table [id=7599d846f23a4be7b3d16137405031b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:42:12.256809 21624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e08ff99b70644764aa98fa3f0a9e8906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:42:12.256842 21758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e08ff99b70644764aa98fa3f0a9e8906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:42:12.257486 21970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:12.245963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:34669 (local address 127.0.0.1:43324)
0618 22:42:12.250249 (+ 4286us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:12.250270 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:42:12.250384 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:42:12.251108 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:42:12.251112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:42:12.251139 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:42:12.251317 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:12.251327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.253027 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.253033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:12.254870 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:12.254887 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.256677 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.256683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:12.256890 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:12.256923 (+ 33us) client_negotiation.cc:769] Sending connection context
0618 22:42:12.257019 (+ 96us) client_negotiation.cc:240] Negotiation successful
0618 22:42:12.257168 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4179,"thread_start_us":4828,"threads_started":1}
I20260618 22:42:12.257675 21972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:12.241693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.5.213.195:34669)
0618 22:42:12.249370 (+ 7677us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:12.249379 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:12.250347 (+ 968us) server_negotiation.cc:407] Connection header received
0618 22:42:12.250570 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:12.250580 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:12.250918 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:12.251116 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:42:12.251472 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.252879 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.255038 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.256401 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.256521 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:12.256753 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:12.257071 (+ 318us) server_negotiation.cc:299] Negotiation successful
0618 22:42:12.257249 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7485,"thread_start_us":7311,"threads_started":1}
I20260618 22:42:12.261813 21891 tablet_service.cc:1467] Processing CreateTablet for tablet e08ff99b70644764aa98fa3f0a9e8906 (DEFAULT_TABLE table=test-table [id=7599d846f23a4be7b3d16137405031b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:42:12.263592 21891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e08ff99b70644764aa98fa3f0a9e8906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:42:12.268996 21974 tablet_bootstrap.cc:492] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df: Bootstrap starting.
I20260618 22:42:12.272223 21975 tablet_bootstrap.cc:492] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a: Bootstrap starting.
I20260618 22:42:12.272430 21974 tablet_bootstrap.cc:654] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df: Neither blocks nor log segments found. Creating new log.
I20260618 22:42:12.273793 21974 log.cc:824] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df: Log is configured to *not* fsync() on all Append() calls
I20260618 22:42:12.276141 21975 tablet_bootstrap.cc:654] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a: Neither blocks nor log segments found. Creating new log.
I20260618 22:42:12.277494 21977 tablet_bootstrap.cc:492] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39: Bootstrap starting.
I20260618 22:42:12.277575 21975 log.cc:824] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:42:12.281072 21974 tablet_bootstrap.cc:492] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df: No bootstrap required, opened a new log
I20260618 22:42:12.281288 21975 tablet_bootstrap.cc:492] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a: No bootstrap required, opened a new log
I20260618 22:42:12.281339 21977 tablet_bootstrap.cc:654] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39: Neither blocks nor log segments found. Creating new log.
I20260618 22:42:12.281355 21974 ts_tablet_manager.cc:1397] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260618 22:42:12.281577 21975 ts_tablet_manager.cc:1397] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260618 22:42:12.282691 21977 log.cc:824] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39: Log is configured to *not* fsync() on all Append() calls
I20260618 22:42:12.287519 21977 tablet_bootstrap.cc:492] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39: No bootstrap required, opened a new log
I20260618 22:42:12.287838 21977 ts_tablet_manager.cc:1397] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260618 22:42:12.289886 21974 raft_consensus.cc:348] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } }
I20260618 22:42:12.290285 21974 raft_consensus.cc:374] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:42:12.290405 21974 raft_consensus.cc:729] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09b731a378054e73831d2313a823a2df, State: Initialized, Role: FOLLOWER
I20260618 22:42:12.291049 21974 consensus_queue.cc:260] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } }
I20260618 22:42:12.291656 21975 raft_consensus.cc:348] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } }
I20260618 22:42:12.292126 21975 raft_consensus.cc:374] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:42:12.292140 21823 heartbeater.cc:503] Master 127.5.213.254:45069 was elected leader, sending a full tablet report...
I20260618 22:42:12.292235 21975 raft_consensus.cc:729] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573d81b018874aa3a49e60a70914643a, State: Initialized, Role: FOLLOWER
I20260618 22:42:12.292810 21974 ts_tablet_manager.cc:1428] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260618 22:42:12.292867 21975 consensus_queue.cc:260] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } }
I20260618 22:42:12.294570 21975 ts_tablet_manager.cc:1428] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
I20260618 22:42:12.298480 21977 raft_consensus.cc:348] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } }
I20260618 22:42:12.299045 21977 raft_consensus.cc:374] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:42:12.299161 21977 raft_consensus.cc:729] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20848997b00049b8a471edcf3a537b39, State: Initialized, Role: FOLLOWER
I20260618 22:42:12.299878 21977 consensus_queue.cc:260] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } }
I20260618 22:42:12.300989 21956 heartbeater.cc:503] Master 127.5.213.254:45069 was elected leader, sending a full tablet report...
I20260618 22:42:12.301635 21977 ts_tablet_manager.cc:1428] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260618 22:42:12.310547 21980 raft_consensus.cc:482] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:42:12.310942 21980 raft_consensus.cc:504] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } }
I20260618 22:42:12.312606 21980 leader_election.cc:288] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20848997b00049b8a471edcf3a537b39 (127.5.213.195:34669), 573d81b018874aa3a49e60a70914643a (127.5.213.193:35929)
I20260618 22:42:12.320109 21972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:12.312908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:49083 (local address 127.5.213.195:34669)
0618 22:42:12.313100 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:12.313107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:12.314025 (+ 918us) server_negotiation.cc:407] Connection header received
0618 22:42:12.314228 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:12.314233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:12.314304 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:12.314368 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:42:12.315176 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.316053 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.318191 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.319092 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.319253 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:12.319310 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:12.319706 (+ 396us) server_negotiation.cc:299] Negotiation successful
0618 22:42:12.319856 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 22:42:12.320132 21971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:12.313284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:50265 (local address 127.5.213.193:35929)
0618 22:42:12.313519 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:12.313530 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:12.314106 (+ 576us) server_negotiation.cc:407] Connection header received
0618 22:42:12.314253 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:12.314256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:12.314315 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:12.314395 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:42:12.315242 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.316157 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.318342 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.319268 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.319460 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:12.319535 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:12.319781 (+ 246us) server_negotiation.cc:299] Negotiation successful
0618 22:42:12.319910 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260618 22:42:12.320482 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:12.312878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:34669 (local address 127.5.213.194:49083)
0618 22:42:12.313903 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:12.313921 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:42:12.314065 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:42:12.314409 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:42:12.314413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:42:12.314436 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:42:12.314999 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:12.315020 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.316170 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.316175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:12.318035 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:12.318048 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.319356 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.319363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:12.319537 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:12.319573 (+ 36us) client_negotiation.cc:769] Sending connection context
0618 22:42:12.319656 (+ 83us) client_negotiation.cc:240] Negotiation successful
0618 22:42:12.319778 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":43,"spinlock_wait_cycles":27520,"thread_start_us":631,"threads_started":1}
I20260618 22:42:12.320505 21984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:12.313058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35929 (local address 127.5.213.194:50265)
0618 22:42:12.313990 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:12.314007 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:42:12.314132 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:42:12.314417 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:42:12.314421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:42:12.314525 (+ 104us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:42:12.315054 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:12.315063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.316326 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.316331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:12.318124 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:12.318139 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.319476 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.319480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:12.319609 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:12.319639 (+ 30us) client_negotiation.cc:769] Sending connection context
0618 22:42:12.319736 (+ 97us) client_negotiation.cc:240] Negotiation successful
0618 22:42:12.319855 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":73,"spinlock_wait_cycles":27520,"thread_start_us":629,"threads_started":1}
I20260618 22:42:12.321736 21911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e08ff99b70644764aa98fa3f0a9e8906" candidate_uuid: "09b731a378054e73831d2313a823a2df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20848997b00049b8a471edcf3a537b39" is_pre_election: true
I20260618 22:42:12.321718 21644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e08ff99b70644764aa98fa3f0a9e8906" candidate_uuid: "09b731a378054e73831d2313a823a2df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573d81b018874aa3a49e60a70914643a" is_pre_election: true
I20260618 22:42:12.322362 21644 raft_consensus.cc:2454] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09b731a378054e73831d2313a823a2df in term 0.
I20260618 22:42:12.322362 21911 raft_consensus.cc:2454] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09b731a378054e73831d2313a823a2df in term 0.
I20260618 22:42:12.323207 21712 leader_election.cc:302] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09b731a378054e73831d2313a823a2df, 573d81b018874aa3a49e60a70914643a; no voters:
I20260618 22:42:12.323633 21980 raft_consensus.cc:2781] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:42:12.323786 21980 raft_consensus.cc:482] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:42:12.323901 21980 raft_consensus.cc:3037] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:42:12.326205 21980 raft_consensus.cc:504] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } }
I20260618 22:42:12.326908 21980 leader_election.cc:288] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [CANDIDATE]: Term 1 election: Requested vote from peers 20848997b00049b8a471edcf3a537b39 (127.5.213.195:34669), 573d81b018874aa3a49e60a70914643a (127.5.213.193:35929)
I20260618 22:42:12.327416 21911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e08ff99b70644764aa98fa3f0a9e8906" candidate_uuid: "09b731a378054e73831d2313a823a2df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20848997b00049b8a471edcf3a537b39"
I20260618 22:42:12.327440 21644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e08ff99b70644764aa98fa3f0a9e8906" candidate_uuid: "09b731a378054e73831d2313a823a2df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573d81b018874aa3a49e60a70914643a"
I20260618 22:42:12.327626 21644 raft_consensus.cc:3037] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:42:12.327641 21911 raft_consensus.cc:3037] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:42:12.329825 21644 raft_consensus.cc:2454] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09b731a378054e73831d2313a823a2df in term 1.
I20260618 22:42:12.329869 21911 raft_consensus.cc:2454] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09b731a378054e73831d2313a823a2df in term 1.
I20260618 22:42:12.330284 21712 leader_election.cc:302] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09b731a378054e73831d2313a823a2df, 20848997b00049b8a471edcf3a537b39, 573d81b018874aa3a49e60a70914643a; no voters:
I20260618 22:42:12.330591 21980 raft_consensus.cc:2781] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:42:12.331063 21980 raft_consensus.cc:686] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [term 1 LEADER]: Becoming Leader. State: Replica: 09b731a378054e73831d2313a823a2df, State: Running, Role: LEADER
I20260618 22:42:12.331527 21980 consensus_queue.cc:237] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } }
I20260618 22:42:12.336459 21499 catalog_manager.cc:5495] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df reported cstate change: term changed from 0 to 1, leader changed from <none> to 09b731a378054e73831d2313a823a2df (127.5.213.194). New cstate: current_term: 1 leader_uuid: "09b731a378054e73831d2313a823a2df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } health_report { overall_health: UNKNOWN } } }
I20260618 22:42:12.356581 21969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:12.351157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35058 (local address 127.5.213.194:40135)
0618 22:42:12.351344 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:12.351349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:12.351366 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:42:12.351562 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:12.351566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:12.351658 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:12.351838 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:42:12.352237 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.352966 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.354361 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.354825 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.354919 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:12.355438 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:42:12.356024 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:42:12.356145 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:12.356273 (+ 128us) server_negotiation.cc:299] Negotiation successful
0618 22:42:12.356366 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 22:42:12.365087 21911 raft_consensus.cc:1261] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39 [term 1 FOLLOWER]: Refusing update from remote peer 09b731a378054e73831d2313a823a2df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:42:12.365087 21644 raft_consensus.cc:1261] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Refusing update from remote peer 09b731a378054e73831d2313a823a2df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:42:12.366063 21985 consensus_queue.cc:1035] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [LEADER]: Connected to new peer: Peer: permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:42:12.366406 21980 consensus_queue.cc:1035] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [LEADER]: Connected to new peer: Peer: permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:42:12.378309 21988 mvcc.cc:205] Tried to move back new op lower bound from 7298345092557066240 to 7298345092433707008. Current Snapshot: MvccSnapshot[applied={T|T < 7298345092557066240}]
I20260618 22:42:12.379850 21989 mvcc.cc:205] Tried to move back new op lower bound from 7298345092557066240 to 7298345092433707008. Current Snapshot: MvccSnapshot[applied={T|T < 7298345092557066240}]
I20260618 22:42:12.393515 21500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:42:12.393812 21500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:42:12.397718 21500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934: Sending DeleteTablet for 3 replicas of tablet e08ff99b70644764aa98fa3f0a9e8906
I20260618 22:42:12.399204 21624 tablet_service.cc:1514] Processing DeleteTablet for tablet e08ff99b70644764aa98fa3f0a9e8906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:42:12 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:45906
I20260618 22:42:12.399209 21758 tablet_service.cc:1514] Processing DeleteTablet for tablet e08ff99b70644764aa98fa3f0a9e8906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:42:12 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:35050
I20260618 22:42:12.400209 21997 tablet_replica.cc:330] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df: stopping tablet replica
I20260618 22:42:12.400229 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 21561
I20260618 22:42:12.400354 21891 tablet_service.cc:1514] Processing DeleteTablet for tablet e08ff99b70644764aa98fa3f0a9e8906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:42:12 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:43324
I20260618 22:42:12.400738 21997 raft_consensus.cc:2229] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:42:12.401476 21997 raft_consensus.cc:2258] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:42:12.404079 21997 ts_tablet_manager.cc:1905] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:42:12.404790 21998 tablet_replica.cc:330] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a: stopping tablet replica
I20260618 22:42:12.405359 21999 tablet_replica.cc:330] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39: stopping tablet replica
I20260618 22:42:12.405309 21998 raft_consensus.cc:2229] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:42:12.406129 21999 raft_consensus.cc:2229] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:42:12.406718 21999 raft_consensus.cc:2258] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:42:12.407882 21998 raft_consensus.cc:2258] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:42:12.410264 21998 ts_tablet_manager.cc:1905] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:42:12.412148 21999 ts_tablet_manager.cc:1905] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:42:12.414149 21997 ts_tablet_manager.cc:1918] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:42:12.414413 21997 log.cc:1196] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/e08ff99b70644764aa98fa3f0a9e8906
I20260618 22:42:12.415077 21997 ts_tablet_manager.cc:1939] T e08ff99b70644764aa98fa3f0a9e8906 P 09b731a378054e73831d2313a823a2df: Deleting consensus metadata
I20260618 22:42:12.417230 21487 catalog_manager.cc:4841] TS 09b731a378054e73831d2313a823a2df (127.5.213.194:40135): tablet e08ff99b70644764aa98fa3f0a9e8906 (table test-table [id=7599d846f23a4be7b3d16137405031b0]) successfully deleted
I20260618 22:42:12.417618 21998 ts_tablet_manager.cc:1918] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:42:12.417879 21998 log.cc:1196] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/e08ff99b70644764aa98fa3f0a9e8906
I20260618 22:42:12.418408 21998 ts_tablet_manager.cc:1939] T e08ff99b70644764aa98fa3f0a9e8906 P 573d81b018874aa3a49e60a70914643a: Deleting consensus metadata
Jun 18 22:42:12 dist-test-slave-lnj3 krb5kdc[21455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:42:12.420365 21485 catalog_manager.cc:4841] TS 573d81b018874aa3a49e60a70914643a (127.5.213.193:35929): tablet e08ff99b70644764aa98fa3f0a9e8906 (table test-table [id=7599d846f23a4be7b3d16137405031b0]) successfully deleted
I20260618 22:42:12.423990 21999 ts_tablet_manager.cc:1918] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:42:12.424162 21999 log.cc:1196] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/e08ff99b70644764aa98fa3f0a9e8906
I20260618 22:42:12.424597 21999 ts_tablet_manager.cc:1939] T e08ff99b70644764aa98fa3f0a9e8906 P 20848997b00049b8a471edcf3a537b39: Deleting consensus metadata
I20260618 22:42:12.426190 21487 catalog_manager.cc:4841] TS 20848997b00049b8a471edcf3a537b39 (127.5.213.195:34669): tablet e08ff99b70644764aa98fa3f0a9e8906 (table test-table [id=7599d846f23a4be7b3d16137405031b0]) successfully deleted
I20260618 22:42:12.429042 21971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:12.401464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45918 (local address 127.5.213.193:35929)
0618 22:42:12.401628 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:12.401633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:12.405075 (+ 3442us) server_negotiation.cc:407] Connection header received
0618 22:42:12.405396 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:12.405402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:12.405488 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:12.405568 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:12.407679 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.408861 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.412363 (+ 3502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.413010 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.413104 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:12.421777 (+ 8673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:12.421794 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:12.421815 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:12.421862 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:12.425881 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:12.426966 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:12.426974 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:12.426976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:12.427111 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:12.427576 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:12.427580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:12.427582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:12.427997 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:12.428230 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:12.428595 (+ 365us) server_negotiation.cc:299] Negotiation successful
0618 22:42:12.428775 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:42:12.430003 21684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:42:12.519168 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 21561
I20260618 22:42:12.534098 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 21695
Jun 18 22:42:12 dist-test-slave-lnj3 krb5kdc[21455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.194@KRBTEST.COM
I20260618 22:42:12.551643 21969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:12.534914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35074 (local address 127.5.213.194:40135)
0618 22:42:12.535200 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:12.535205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:12.535232 (+ 27us) server_negotiation.cc:407] Connection header received
0618 22:42:12.535300 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:12.535305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:12.535398 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:12.535503 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:12.536860 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.538090 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.539474 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.540143 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.540270 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:12.544616 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:12.544645 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:12.544667 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:12.544727 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:12.548554 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:12.549339 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:12.549350 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:12.549354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:12.549549 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:12.549942 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:12.549947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:12.549949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:12.550423 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:12.550807 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:12.551012 (+ 205us) server_negotiation.cc:299] Negotiation successful
0618 22:42:12.551246 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260618 22:42:12.552428 21818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:42:12.674933 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 21695
I20260618 22:42:12.689934 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 21828
Jun 18 22:42:12 dist-test-slave-lnj3 krb5kdc[21455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.195@KRBTEST.COM
I20260618 22:42:12.704483 21972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:12.690594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43326 (local address 127.5.213.195:34669)
0618 22:42:12.690814 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:12.690819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:12.690854 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:42:12.691008 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:12.691012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:12.691076 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:12.691169 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:12.692341 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.693064 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.694347 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:12.694769 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:12.694857 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:12.699306 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:12.699320 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:12.699336 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:12.699373 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:12.701772 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:12.702397 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:12.702404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:12.702407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:12.702538 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:12.703012 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:12.703017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:12.703019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:12.703382 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:12.703651 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:12.703982 (+ 331us) server_negotiation.cc:299] Negotiation successful
0618 22:42:12.704161 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260618 22:42:12.705097 21951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:42:12.792194 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 21828
I20260618 22:42:12.805208 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 21468
I20260618 22:42:12.806290 21530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:42:12.914170 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 21468
I20260618 22:42:32.931501 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:45069
--webserver_interface=127.5.213.254
--webserver_port=36885
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:45069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:42:33.249421 22005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:42:33.249747 22005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:42:33.249889 22005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:42:33.257195 22005 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260618 22:42:33.257308 22005 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260618 22:42:33.257367 22005 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260618 22:42:33.257433 22005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:42:33.257507 22005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:42:33.257598 22005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:42:33.257670 22005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:42:33.257745 22005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:42:33.268854 22005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:45069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:45069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=36885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:42:33.270190 22005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:42:33.271549 22005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:42:33.293510 22011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:33.293486 22012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:42:33.293723 22005 server_base.cc:1034] running on GCE node
W20260618 22:42:33.293437 22014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:42:33.295104 22005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:42:33.296864 22005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:42:33.298137 22005 hybrid_clock.cc:648] HybridClock initialized: now 1781822553298056 us; error 53 us; skew 500 ppm
I20260618 22:42:33.298575 22005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:42:33 dist-test-slave-lnj3 krb5kdc[21455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822553, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:42:33.312650 22005 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:42:33.314740 22005 webserver.cc:460] Webserver started at http://127.5.213.254:36885/ using document root <none> and password file <none>
I20260618 22:42:33.315579 22005 fs_manager.cc:359] Metadata directory not provided
I20260618 22:42:33.315708 22005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:42:33.322187 22005 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260618 22:42:33.325512 22021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:33.327082 22005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260618 22:42:33.327246 22005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "9c06746ac2c24833879dde032c0ac934"
format_stamp: "Formatted at 2026-06-18 22:42:10 on dist-test-slave-lnj3"
server_key: "4f0ec6b4daa8cfce989761822a24395e"
server_key_iv: "d6dbb391f4db6184aee08c9ccf6fbc3e"
server_key_version: "encryptionkey@0"
I20260618 22:42:33.328328 22005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:42:33.349838 22005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:42:33.354866 22005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:42:33.355197 22005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:42:33.373348 22005 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:45069
I20260618 22:42:33.373390 22072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:45069 every 8 connection(s)
I20260618 22:42:33.375036 22005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:42:33.382603 22073 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:42:33.384009 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 22005
I20260618 22:42:33.384781 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:35929
--local_ip_for_outbound_sockets=127.5.213.193
--tserver_master_addrs=127.5.213.254:45069
--webserver_port=37697
--webserver_interface=127.5.213.193
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:42:33.386459 22073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934: Bootstrap starting.
I20260618 22:42:33.407262 22073 log.cc:824] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934: Log is configured to *not* fsync() on all Append() calls
I20260618 22:42:33.419859 22073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:42:33.420379 22073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934: Bootstrap complete.
I20260618 22:42:33.429507 22073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c06746ac2c24833879dde032c0ac934" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 45069 } }
I20260618 22:42:33.430393 22073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c06746ac2c24833879dde032c0ac934, State: Initialized, Role: FOLLOWER
I20260618 22:42:33.431084 22073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c06746ac2c24833879dde032c0ac934" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 45069 } }
I20260618 22:42:33.431324 22073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:42:33.431452 22073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:42:33.431638 22073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 1 FOLLOWER]: Advancing to term 2
I20260618 22:42:33.435853 22073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c06746ac2c24833879dde032c0ac934" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 45069 } }
I20260618 22:42:33.436416 22073 leader_election.cc:302] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c06746ac2c24833879dde032c0ac934; no voters:
I20260618 22:42:33.437000 22073 leader_election.cc:288] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260618 22:42:33.437187 22077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 2 FOLLOWER]: Leader election won for term 2
I20260618 22:42:33.438122 22077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 2 LEADER]: Becoming Leader. State: Replica: 9c06746ac2c24833879dde032c0ac934, State: Running, Role: LEADER
I20260618 22:42:33.438783 22077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c06746ac2c24833879dde032c0ac934" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 45069 } }
I20260618 22:42:33.439030 22073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:42:33.441186 22078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9c06746ac2c24833879dde032c0ac934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c06746ac2c24833879dde032c0ac934" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 45069 } } }
I20260618 22:42:33.441540 22078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [sys.catalog]: This master's current role is: LEADER
I20260618 22:42:33.443795 22079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c06746ac2c24833879dde032c0ac934. Latest consensus state: current_term: 2 leader_uuid: "9c06746ac2c24833879dde032c0ac934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c06746ac2c24833879dde032c0ac934" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 45069 } } }
I20260618 22:42:33.444165 22079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [sys.catalog]: This master's current role is: LEADER
I20260618 22:42:33.445684 22082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:42:33.453873 22082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:42:33.455442 22082 catalog_manager.cc:1259] Loaded cluster ID: 7b998e51a5fb427491a1a0ade82e0a8c
I20260618 22:42:33.455624 22082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:42:33.459834 22082 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:42:33.461675 22082 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934: Loaded TSK: 0
I20260618 22:42:33.481161 22094 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260618 22:42:33.494076 22082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934: Generated new TSK 1
I20260618 22:42:33.494535 22082 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 22:42:33.727844 22075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:42:33.728132 22075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:42:33.728248 22075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:42:33.735757 22075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:42:33.735873 22075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:42:33.736016 22075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:42:33.746044 22075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:35929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=37697
--tserver_master_addrs=127.5.213.254:45069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:42:33.747512 22075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:42:33.748930 22075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:42:33.764799 22101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:33.765021 22103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:33.765193 22100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:42:33.765849 22075 server_base.cc:1034] running on GCE node
I20260618 22:42:33.766471 22075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:42:33.767551 22075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:42:33.768823 22075 hybrid_clock.cc:648] HybridClock initialized: now 1781822553768728 us; error 43 us; skew 500 ppm
I20260618 22:42:33.769146 22075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:42:33 dist-test-slave-lnj3 krb5kdc[21455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822553, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:42:33.772281 22075 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:42:33.773845 22075 webserver.cc:460] Webserver started at http://127.5.213.193:37697/ using document root <none> and password file <none>
I20260618 22:42:33.774554 22075 fs_manager.cc:359] Metadata directory not provided
I20260618 22:42:33.774734 22075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:42:33.780297 22075 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260618 22:42:33.783056 22110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:33.784411 22075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260618 22:42:33.784564 22075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "573d81b018874aa3a49e60a70914643a"
format_stamp: "Formatted at 2026-06-18 22:42:11 on dist-test-slave-lnj3"
server_key: "d970bca08826606cebdc5be6c8c3d04e"
server_key_iv: "77c5d1e46d0f92258e064582dd2fc8ca"
server_key_version: "encryptionkey@0"
I20260618 22:42:33.785516 22075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:42:33.826771 22075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:42:33.831107 22075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:42:33.831365 22075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:42:33.832419 22075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:42:33.833818 22075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:42:33.833907 22075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:33.834008 22075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:42:33.834081 22075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:33.865301 22075 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:35929
I20260618 22:42:33.865396 22222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:35929 every 8 connection(s)
I20260618 22:42:33.866815 22075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:42:33.868750 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 22075
I20260618 22:42:33.869463 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:40135
--local_ip_for_outbound_sockets=127.5.213.194
--tserver_master_addrs=127.5.213.254:45069
--webserver_port=38561
--webserver_interface=127.5.213.194
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:42:33 dist-test-slave-lnj3 krb5kdc[21455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822553, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:42:33.892391 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:33.869711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:58367 (local address 127.5.213.254:45069)
0618 22:42:33.870272 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:33.870285 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:33.871076 (+ 791us) server_negotiation.cc:407] Connection header received
0618 22:42:33.873418 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:33.873432 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:33.873808 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:33.874150 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:33.875766 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:33.877020 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:33.879252 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:33.879959 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:33.880111 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:33.884196 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:33.884204 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:33.884222 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:33.884269 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:33.887270 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:33.888357 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:33.888364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:33.888371 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:33.888567 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:33.889070 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:33.889073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:33.889075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:33.889583 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:33.889781 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:33.890709 (+ 928us) server_negotiation.cc:299] Negotiation successful
0618 22:42:33.891120 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":227,"threads_started":1}
I20260618 22:42:33.892392 22226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:33.870086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.193:58367)
0618 22:42:33.870900 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:33.870944 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:42:33.872828 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:42:33.874327 (+ 1499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:42:33.874338 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:42:33.874999 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:42:33.875553 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:33.875571 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:33.877193 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:33.877198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:33.879092 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:33.879107 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:33.880188 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:33.880195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:33.880452 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:33.881179 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:42:33.881207 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:42:33.883882 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:42:33.887441 (+ 3559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:42:33.887452 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:42:33.887470 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:42:33.888180 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:42:33.888688 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:42:33.888693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:42:33.888697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:42:33.888956 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:42:33.889738 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:42:33.889747 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:42:33.890145 (+ 398us) client_negotiation.cc:769] Sending connection context
0618 22:42:33.890410 (+ 265us) client_negotiation.cc:240] Negotiation successful
0618 22:42:33.891047 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":295,"threads_started":1}
I20260618 22:42:33.895471 22223 heartbeater.cc:347] Connected to a master server at 127.5.213.254:45069
I20260618 22:42:33.895781 22223 heartbeater.cc:464] Registering TS with master...
I20260618 22:42:33.896556 22223 heartbeater.cc:511] Master 127.5.213.254:45069 requested a full tablet report, sending...
I20260618 22:42:33.898777 22038 ts_manager.cc:194] Registered new tserver with Master: 573d81b018874aa3a49e60a70914643a (127.5.213.193:35929)
I20260618 22:42:33.900300 22038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:58367
W20260618 22:42:34.194612 22227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:42:34.194963 22227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:42:34.195107 22227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:42:34.202376 22227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:42:34.202502 22227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:42:34.202667 22227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:42:34.212217 22227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:40135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=38561
--tserver_master_addrs=127.5.213.254:45069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:42:34.213475 22227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:42:34.214773 22227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:42:34.231117 22232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:34.231082 22235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:34.231046 22233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:42:34.232116 22227 server_base.cc:1034] running on GCE node
I20260618 22:42:34.232790 22227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:42:34.233777 22227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:42:34.234967 22227 hybrid_clock.cc:648] HybridClock initialized: now 1781822554234916 us; error 58 us; skew 500 ppm
I20260618 22:42:34.235265 22227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:42:34 dist-test-slave-lnj3 krb5kdc[21455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822554, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:42:34.238263 22227 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:42:34.239835 22227 webserver.cc:460] Webserver started at http://127.5.213.194:38561/ using document root <none> and password file <none>
I20260618 22:42:34.240561 22227 fs_manager.cc:359] Metadata directory not provided
I20260618 22:42:34.240702 22227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:42:34.246214 22227 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260618 22:42:34.249085 22242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:34.250495 22227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260618 22:42:34.250669 22227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "09b731a378054e73831d2313a823a2df"
format_stamp: "Formatted at 2026-06-18 22:42:11 on dist-test-slave-lnj3"
server_key: "ac955d938b10fc03cf0ad3f4c4b84024"
server_key_iv: "6a497b162b618903fef4ea74e81a09fe"
server_key_version: "encryptionkey@0"
I20260618 22:42:34.251606 22227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:42:34.270035 22227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:42:34.274202 22227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:42:34.274452 22227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:42:34.275523 22227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:42:34.276831 22227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:42:34.276925 22227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:34.277031 22227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:42:34.277108 22227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:34.310333 22227 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:40135
I20260618 22:42:34.310402 22354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:40135 every 8 connection(s)
I20260618 22:42:34.311861 22227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:42:34.314036 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 22227
I20260618 22:42:34.314718 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:34669
--local_ip_for_outbound_sockets=127.5.213.195
--tserver_master_addrs=127.5.213.254:45069
--webserver_port=43319
--webserver_interface=127.5.213.195
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:42:34 dist-test-slave-lnj3 krb5kdc[21455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822554, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:42:34.334002 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:34.314522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:54361 (local address 127.5.213.254:45069)
0618 22:42:34.314754 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:34.314761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:34.316069 (+ 1308us) server_negotiation.cc:407] Connection header received
0618 22:42:34.318385 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:34.318389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:34.318459 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:34.318545 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:34.320512 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.321230 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.323248 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.323882 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.324006 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:34.327649 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:34.327655 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:34.327659 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:34.327697 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:34.330676 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:34.331385 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:34.331390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:34.331394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:34.331542 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:34.332032 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:34.332036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:34.332039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:34.332393 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:34.332535 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:34.333385 (+ 850us) server_negotiation.cc:299] Negotiation successful
0618 22:42:34.333638 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260618 22:42:34.334841 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:34.315101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.194:54361)
0618 22:42:34.315891 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:34.315940 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0618 22:42:34.318079 (+ 2139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:42:34.318837 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:42:34.318848 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:42:34.319558 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:42:34.320334 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:34.320357 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.321373 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.321378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:34.323088 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:34.323101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.324066 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.324071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:34.324301 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:34.324951 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:42:34.324973 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:42:34.327402 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:42:34.330812 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:42:34.330819 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:42:34.330832 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:42:34.331266 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:42:34.331699 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:42:34.331706 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:42:34.331709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:42:34.331905 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:42:34.332536 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:42:34.332545 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:42:34.333008 (+ 463us) client_negotiation.cc:769] Sending connection context
0618 22:42:34.333265 (+ 257us) client_negotiation.cc:240] Negotiation successful
0618 22:42:34.333685 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":282,"threads_started":1}
I20260618 22:42:34.336357 22355 heartbeater.cc:347] Connected to a master server at 127.5.213.254:45069
I20260618 22:42:34.336725 22355 heartbeater.cc:464] Registering TS with master...
I20260618 22:42:34.337649 22355 heartbeater.cc:511] Master 127.5.213.254:45069 requested a full tablet report, sending...
I20260618 22:42:34.339329 22038 ts_manager.cc:194] Registered new tserver with Master: 09b731a378054e73831d2313a823a2df (127.5.213.194:40135)
I20260618 22:42:34.340173 22038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:54361
W20260618 22:42:34.644186 22358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:42:34.644479 22358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:42:34.644670 22358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:42:34.652021 22358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:42:34.652149 22358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:42:34.652300 22358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:42:34.662122 22358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:34669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=43319
--tserver_master_addrs=127.5.213.254:45069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:42:34.663427 22358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:42:34.664734 22358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:42:34.677481 22364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:34.677503 22366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:34.677577 22363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:42:34.678228 22358 server_base.cc:1034] running on GCE node
I20260618 22:42:34.678896 22358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:42:34.679817 22358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:42:34.681020 22358 hybrid_clock.cc:648] HybridClock initialized: now 1781822554680957 us; error 53 us; skew 500 ppm
I20260618 22:42:34.681382 22358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:42:34 dist-test-slave-lnj3 krb5kdc[21455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822554, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:42:34.684417 22358 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:42:34.685878 22358 webserver.cc:460] Webserver started at http://127.5.213.195:43319/ using document root <none> and password file <none>
I20260618 22:42:34.686517 22358 fs_manager.cc:359] Metadata directory not provided
I20260618 22:42:34.686676 22358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:42:34.692042 22358 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:42:34.694572 22373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:34.695920 22358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:42:34.696062 22358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "20848997b00049b8a471edcf3a537b39"
format_stamp: "Formatted at 2026-06-18 22:42:12 on dist-test-slave-lnj3"
server_key: "553c035740e5989fe68d908038d4840c"
server_key_iv: "73d7b5f1aec13a420a56553bc2c84df6"
server_key_version: "encryptionkey@0"
I20260618 22:42:34.696988 22358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:42:34.708956 22358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:42:34.712891 22358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:42:34.713148 22358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:42:34.714073 22358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:42:34.715386 22358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:42:34.715468 22358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:34.715569 22358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:42:34.715626 22358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:34.745970 22358 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:34669
I20260618 22:42:34.746016 22485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:34669 every 8 connection(s)
I20260618 22:42:34.747490 22358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:42:34.750382 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 22358
Jun 18 22:42:34 dist-test-slave-lnj3 krb5kdc[21455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822554, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:42:34.767256 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:34.751220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60124 (local address 127.5.213.254:45069)
0618 22:42:34.754456 (+ 3236us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:34.754461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:34.754478 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:42:34.754525 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:34.754530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:34.754593 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:34.754726 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:34.756310 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.757097 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.758437 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.758926 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.759029 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:34.761825 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:34.761830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:34.761833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:34.761866 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:34.764718 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:34.765335 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:34.765341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:34.765343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:34.765471 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:34.765834 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:34.765839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:34.765842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:34.766122 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:34.766373 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:34.766668 (+ 295us) server_negotiation.cc:299] Negotiation successful
0618 22:42:34.766880 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3126,"thread_start_us":2997,"threads_started":1}
I20260618 22:42:34.767998 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:34.750097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:60013 (local address 127.5.213.254:45069)
0618 22:42:34.750319 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:34.750326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:34.751523 (+ 1197us) server_negotiation.cc:407] Connection header received
0618 22:42:34.753418 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:34.753424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:34.753513 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:34.753611 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:34.755702 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.756491 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.758034 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.758606 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.758773 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:34.762365 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:34.762370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:34.762373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:34.762413 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:34.765089 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:34.765779 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:34.765783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:34.765785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:34.765897 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:34.766315 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:34.766320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:34.766323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:34.766580 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:34.766744 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:34.767545 (+ 801us) server_negotiation.cc:299] Negotiation successful
0618 22:42:34.767726 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260618 22:42:34.769302 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:34.750569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.195:60013)
0618 22:42:34.751374 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:34.751420 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 22:42:34.753146 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:42:34.754046 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:42:34.754059 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:42:34.754919 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:42:34.755526 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:34.755544 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.756659 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.756662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:34.757897 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:34.757908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.758856 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.758864 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:34.759115 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:34.759778 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:42:34.759812 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:42:34.762168 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:42:34.765213 (+ 3045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:42:34.765223 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:42:34.765241 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:42:34.765661 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:42:34.765998 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:42:34.766003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:42:34.766007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:42:34.766199 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:42:34.766748 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:42:34.766757 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:42:34.767101 (+ 344us) client_negotiation.cc:769] Sending connection context
0618 22:42:34.767468 (+ 367us) client_negotiation.cc:240] Negotiation successful
0618 22:42:34.768010 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":304,"threads_started":1}
I20260618 22:42:34.770764 22486 heartbeater.cc:347] Connected to a master server at 127.5.213.254:45069
I20260618 22:42:34.771023 22486 heartbeater.cc:464] Registering TS with master...
I20260618 22:42:34.771670 22486 heartbeater.cc:511] Master 127.5.213.254:45069 requested a full tablet report, sending...
I20260618 22:42:34.773010 22038 ts_manager.cc:194] Registered new tserver with Master: 20848997b00049b8a471edcf3a537b39 (127.5.213.195:34669)
I20260618 22:42:34.773785 22038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:60013
I20260618 22:42:34.782367 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:42:34.789055 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:34.783385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60132 (local address 127.5.213.254:45069)
0618 22:42:34.783532 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:34.783537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:34.783979 (+ 442us) server_negotiation.cc:407] Connection header received
0618 22:42:34.784189 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:34.784200 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:34.784360 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:34.784484 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:42:34.784924 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.785688 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.787293 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.787691 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.787798 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:34.788310 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:42:34.788517 (+ 207us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0618 22:42:34.788854 (+ 337us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60132: token expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260618 22:42:34.789179 22225 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60132: token expired
W20260618 22:42:34.789758 21961 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.5.213.254:45069)
I20260618 22:42:34.801681 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:34.790197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.5.213.254:45069)
0618 22:42:34.790344 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:34.790348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:34.790363 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:42:34.790441 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:34.790444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:34.790501 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:34.790576 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:34.791872 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.792571 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.793758 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.794136 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.794227 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:34.796833 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:34.796837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:34.796840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:34.796873 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:34.799224 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:34.799814 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:34.799817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:34.799819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:34.799930 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:34.800331 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:34.800334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:34.800336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:34.800669 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:34.800849 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:34.801242 (+ 393us) server_negotiation.cc:299] Negotiation successful
0618 22:42:34.801404 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:42:34.805015 22038 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:42:34.812002 22038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:42:34.846825 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:34.823792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:40135 (local address 127.0.0.1:59664)
0618 22:42:34.824698 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:34.824733 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0618 22:42:34.824889 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:42:34.840074 (+ 15185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:42:34.840081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:42:34.840111 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:42:34.840597 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:34.840613 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.842173 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.842178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:34.844348 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:34.844360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.845738 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.845744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:34.845999 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:34.846056 (+ 57us) client_negotiation.cc:769] Sending connection context
0618 22:42:34.846174 (+ 118us) client_negotiation.cc:240] Negotiation successful
0618 22:42:34.846364 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":77,"thread_start_us":555,"threads_started":1}
I20260618 22:42:34.847641 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:34.823926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.5.213.194:40135)
0618 22:42:34.839416 (+ 15490us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:34.839424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:34.839493 (+ 69us) server_negotiation.cc:407] Connection header received
0618 22:42:34.839577 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:34.839583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:34.839855 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:34.840048 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:42:34.840849 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.842050 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.844498 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.845525 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.845663 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:34.845865 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:34.846241 (+ 376us) server_negotiation.cc:299] Negotiation successful
0618 22:42:34.846451 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15326,"thread_start_us":15241,"threads_started":1}
I20260618 22:42:34.847811 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:34.823894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35929 (local address 127.0.0.1:44732)
0618 22:42:34.836212 (+ 12318us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:34.836232 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:42:34.836381 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:42:34.840583 (+ 4202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:42:34.840587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:42:34.840607 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:42:34.841020 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:34.841030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.843017 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.843022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:34.845107 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:34.845121 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.847017 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.847024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:34.847194 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:34.847238 (+ 44us) client_negotiation.cc:769] Sending connection context
0618 22:42:34.847302 (+ 64us) client_negotiation.cc:240] Negotiation successful
0618 22:42:34.847423 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12175,"mutex_wait_us":5,"thread_start_us":12043,"threads_started":1}
I20260618 22:42:34.848750 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:34.824427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44732 (local address 127.5.213.193:35929)
0618 22:42:34.839879 (+ 15452us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:34.839886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:34.839908 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:42:34.839980 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:34.839986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:34.840234 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:34.840534 (+ 300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:42:34.841220 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.842873 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.845277 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.846766 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.848020 (+ 1254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:34.848185 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:34.848254 (+ 69us) server_negotiation.cc:299] Negotiation successful
0618 22:42:34.848419 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15257,"thread_start_us":15140,"threads_started":1}
I20260618 22:42:34.849074 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:34.825062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:34669 (local address 127.0.0.1:37596)
0618 22:42:34.839127 (+ 14065us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:34.839145 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:42:34.839357 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:42:34.840363 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:42:34.840367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:42:34.840391 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:42:34.840818 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:34.840829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.844174 (+ 3345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.844180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:34.845969 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:34.845982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.848304 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.848310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:34.848498 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:34.848532 (+ 34us) client_negotiation.cc:769] Sending connection context
0618 22:42:34.848614 (+ 82us) client_negotiation.cc:240] Negotiation successful
0618 22:42:34.848737 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13940,"mutex_wait_us":2,"thread_start_us":13808,"threads_started":1}
I20260618 22:42:34.849546 22501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:34.825496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37596 (local address 127.5.213.195:34669)
0618 22:42:34.838443 (+ 12947us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:34.838455 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:34.839249 (+ 794us) server_negotiation.cc:407] Connection header received
0618 22:42:34.839546 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:34.839555 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:34.839850 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:34.840035 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:42:34.841218 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.842867 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.846125 (+ 3258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.847829 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.847945 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:34.848175 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:34.849058 (+ 883us) server_negotiation.cc:299] Negotiation successful
0618 22:42:34.849219 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12739,"thread_start_us":12965,"threads_started":1}
I20260618 22:42:34.851353 22290 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac6e9d3925949e4ab31775927bd2104 (DEFAULT_TABLE table=test-table [id=b53b25c8204c45ce93e1abb40a73215d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:42:34.852485 22290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ac6e9d3925949e4ab31775927bd2104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:42:34.852173 22158 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac6e9d3925949e4ab31775927bd2104 (DEFAULT_TABLE table=test-table [id=b53b25c8204c45ce93e1abb40a73215d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:42:34.852860 22421 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac6e9d3925949e4ab31775927bd2104 (DEFAULT_TABLE table=test-table [id=b53b25c8204c45ce93e1abb40a73215d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:42:34.853523 22158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ac6e9d3925949e4ab31775927bd2104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:42:34.854184 22421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ac6e9d3925949e4ab31775927bd2104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:42:34.864400 22502 tablet_bootstrap.cc:492] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df: Bootstrap starting.
I20260618 22:42:34.869029 22503 tablet_bootstrap.cc:492] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a: Bootstrap starting.
I20260618 22:42:34.870805 22504 tablet_bootstrap.cc:492] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39: Bootstrap starting.
I20260618 22:42:34.872785 22503 tablet_bootstrap.cc:654] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a: Neither blocks nor log segments found. Creating new log.
I20260618 22:42:34.874128 22503 log.cc:824] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:42:34.874964 22504 tablet_bootstrap.cc:654] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39: Neither blocks nor log segments found. Creating new log.
I20260618 22:42:34.874965 22502 tablet_bootstrap.cc:654] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df: Neither blocks nor log segments found. Creating new log.
I20260618 22:42:34.876183 22504 log.cc:824] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39: Log is configured to *not* fsync() on all Append() calls
I20260618 22:42:34.876212 22502 log.cc:824] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df: Log is configured to *not* fsync() on all Append() calls
I20260618 22:42:34.880107 22503 tablet_bootstrap.cc:492] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a: No bootstrap required, opened a new log
I20260618 22:42:34.880342 22504 tablet_bootstrap.cc:492] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39: No bootstrap required, opened a new log
I20260618 22:42:34.880344 22502 tablet_bootstrap.cc:492] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df: No bootstrap required, opened a new log
I20260618 22:42:34.880369 22503 ts_tablet_manager.cc:1397] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20260618 22:42:34.880606 22504 ts_tablet_manager.cc:1397] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260618 22:42:34.880605 22502 ts_tablet_manager.cc:1397] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20260618 22:42:34.888051 22503 raft_consensus.cc:348] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } }
I20260618 22:42:34.888479 22503 raft_consensus.cc:374] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:42:34.888602 22503 raft_consensus.cc:729] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573d81b018874aa3a49e60a70914643a, State: Initialized, Role: FOLLOWER
I20260618 22:42:34.889189 22503 consensus_queue.cc:260] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } }
I20260618 22:42:34.890327 22223 heartbeater.cc:503] Master 127.5.213.254:45069 was elected leader, sending a full tablet report...
I20260618 22:42:34.890935 22503 ts_tablet_manager.cc:1428] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a: Time spent starting tablet: real 0.010s user 0.005s sys 0.007s
I20260618 22:42:34.890822 22502 raft_consensus.cc:348] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } }
I20260618 22:42:34.891252 22502 raft_consensus.cc:374] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:42:34.891359 22502 raft_consensus.cc:729] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09b731a378054e73831d2313a823a2df, State: Initialized, Role: FOLLOWER
I20260618 22:42:34.891485 22504 raft_consensus.cc:348] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } }
I20260618 22:42:34.891889 22504 raft_consensus.cc:374] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:42:34.891981 22504 raft_consensus.cc:729] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20848997b00049b8a471edcf3a537b39, State: Initialized, Role: FOLLOWER
I20260618 22:42:34.892012 22502 consensus_queue.cc:260] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } }
I20260618 22:42:34.892551 22504 consensus_queue.cc:260] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } }
I20260618 22:42:34.893108 22355 heartbeater.cc:503] Master 127.5.213.254:45069 was elected leader, sending a full tablet report...
I20260618 22:42:34.893826 22502 ts_tablet_manager.cc:1428] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df: Time spent starting tablet: real 0.013s user 0.001s sys 0.013s
I20260618 22:42:34.895375 22486 heartbeater.cc:503] Master 127.5.213.254:45069 was elected leader, sending a full tablet report...
I20260618 22:42:34.895910 22504 ts_tablet_manager.cc:1428] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39: Time spent starting tablet: real 0.015s user 0.007s sys 0.008s
I20260618 22:42:34.936120 22510 raft_consensus.cc:482] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:42:34.936411 22510 raft_consensus.cc:504] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } }
I20260618 22:42:34.938108 22510 leader_election.cc:288] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09b731a378054e73831d2313a823a2df (127.5.213.194:40135), 573d81b018874aa3a49e60a70914643a (127.5.213.193:35929)
I20260618 22:42:34.944312 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:34.938502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35929 (local address 127.5.213.195:45667)
0618 22:42:34.939002 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:34.939023 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:42:34.939172 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:42:34.939520 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:42:34.939525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:42:34.939552 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:42:34.939736 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:34.939749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.940904 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.940909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:34.942444 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:34.942455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.943628 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.943633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:34.943813 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:34.943848 (+ 35us) client_negotiation.cc:769] Sending connection context
0618 22:42:34.943936 (+ 88us) client_negotiation.cc:240] Negotiation successful
0618 22:42:34.944029 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":53}
I20260618 22:42:34.944379 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:34.938785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:45667 (local address 127.5.213.193:35929)
0618 22:42:34.938971 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:34.938976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:34.939108 (+ 132us) server_negotiation.cc:407] Connection header received
0618 22:42:34.939314 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:34.939320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:34.939403 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:34.939491 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:42:34.939881 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.940784 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.942593 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.943487 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.943602 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:34.943657 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:34.943975 (+ 318us) server_negotiation.cc:299] Negotiation successful
0618 22:42:34.944098 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:42:34.944483 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:34.938538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:43209 (local address 127.5.213.194:40135)
0618 22:42:34.938787 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:34.938793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:34.939440 (+ 647us) server_negotiation.cc:407] Connection header received
0618 22:42:34.939595 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:34.939600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:34.939665 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:34.939802 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:42:34.940372 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.941179 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.942879 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.943656 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.943798 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:34.943867 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:34.944153 (+ 286us) server_negotiation.cc:299] Negotiation successful
0618 22:42:34.944268 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260618 22:42:34.944602 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:34.938551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:40135 (local address 127.5.213.195:43209)
0618 22:42:34.939337 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:34.939353 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:42:34.939478 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:42:34.939837 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:42:34.939841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:42:34.939865 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:42:34.940222 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:34.940234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.941308 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.941312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:34.942743 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:34.942757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.943847 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.943851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:34.944002 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:34.944034 (+ 32us) client_negotiation.cc:769] Sending connection context
0618 22:42:34.944115 (+ 81us) client_negotiation.cc:240] Negotiation successful
0618 22:42:34.944235 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":2,"thread_start_us":455,"threads_started":1}
I20260618 22:42:34.945495 22310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac6e9d3925949e4ab31775927bd2104" candidate_uuid: "20848997b00049b8a471edcf3a537b39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09b731a378054e73831d2313a823a2df" is_pre_election: true
I20260618 22:42:34.945474 22178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac6e9d3925949e4ab31775927bd2104" candidate_uuid: "20848997b00049b8a471edcf3a537b39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573d81b018874aa3a49e60a70914643a" is_pre_election: true
I20260618 22:42:34.945933 22310 raft_consensus.cc:2454] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20848997b00049b8a471edcf3a537b39 in term 0.
I20260618 22:42:34.945933 22178 raft_consensus.cc:2454] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20848997b00049b8a471edcf3a537b39 in term 0.
I20260618 22:42:34.946561 22377 leader_election.cc:302] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09b731a378054e73831d2313a823a2df, 20848997b00049b8a471edcf3a537b39; no voters:
I20260618 22:42:34.947001 22510 raft_consensus.cc:2781] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:42:34.947166 22510 raft_consensus.cc:482] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:42:34.947294 22510 raft_consensus.cc:3037] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:42:34.949419 22510 raft_consensus.cc:504] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } }
I20260618 22:42:34.949983 22510 leader_election.cc:288] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [CANDIDATE]: Term 1 election: Requested vote from peers 09b731a378054e73831d2313a823a2df (127.5.213.194:40135), 573d81b018874aa3a49e60a70914643a (127.5.213.193:35929)
I20260618 22:42:34.950434 22310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac6e9d3925949e4ab31775927bd2104" candidate_uuid: "20848997b00049b8a471edcf3a537b39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09b731a378054e73831d2313a823a2df"
I20260618 22:42:34.950477 22178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac6e9d3925949e4ab31775927bd2104" candidate_uuid: "20848997b00049b8a471edcf3a537b39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573d81b018874aa3a49e60a70914643a"
I20260618 22:42:34.950683 22310 raft_consensus.cc:3037] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:42:34.950695 22178 raft_consensus.cc:3037] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:42:34.953783 22310 raft_consensus.cc:2454] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20848997b00049b8a471edcf3a537b39 in term 1.
I20260618 22:42:34.953779 22178 raft_consensus.cc:2454] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20848997b00049b8a471edcf3a537b39 in term 1.
I20260618 22:42:34.954212 22375 leader_election.cc:302] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20848997b00049b8a471edcf3a537b39, 573d81b018874aa3a49e60a70914643a; no voters:
I20260618 22:42:34.954493 22510 raft_consensus.cc:2781] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:42:34.954952 22510 raft_consensus.cc:686] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [term 1 LEADER]: Becoming Leader. State: Replica: 20848997b00049b8a471edcf3a537b39, State: Running, Role: LEADER
I20260618 22:42:34.955454 22510 consensus_queue.cc:237] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } }
I20260618 22:42:34.960300 22038 catalog_manager.cc:5495] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20848997b00049b8a471edcf3a537b39 (127.5.213.195). New cstate: current_term: 1 leader_uuid: "20848997b00049b8a471edcf3a537b39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } health_report { overall_health: HEALTHY } } }
I20260618 22:42:34.988981 22501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:34.983879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.5.213.195:34669)
0618 22:42:34.984046 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:34.984051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:34.984069 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:42:34.984206 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:34.984210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:34.984286 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:34.984404 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:42:34.984739 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.985440 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.986898 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:34.987331 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:34.987420 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:34.987923 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:42:34.988454 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:42:34.988555 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:34.988723 (+ 168us) server_negotiation.cc:299] Negotiation successful
0618 22:42:34.988784 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:42:34.996762 22310 raft_consensus.cc:1261] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df [term 1 FOLLOWER]: Refusing update from remote peer 20848997b00049b8a471edcf3a537b39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:42:34.996790 22178 raft_consensus.cc:1261] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Refusing update from remote peer 20848997b00049b8a471edcf3a537b39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:42:34.997684 22512 consensus_queue.cc:1035] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:42:34.998003 22510 consensus_queue.cc:1035] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260618 22:42:34.998881 22487 tablet.cc:2369] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:42:35.023180 22038 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:42:35.023402 22038 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:42:35.026618 22038 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934: Sending DeleteTablet for 3 replicas of tablet 0ac6e9d3925949e4ab31775927bd2104
I20260618 22:42:35.027871 22290 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ac6e9d3925949e4ab31775927bd2104 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:42:35 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:59664
I20260618 22:42:35.028036 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 22075
I20260618 22:42:35.027824 22158 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ac6e9d3925949e4ab31775927bd2104 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:42:35 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:44732
I20260618 22:42:35.028743 22421 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ac6e9d3925949e4ab31775927bd2104 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:42:35 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:37596
I20260618 22:42:35.039672 22526 tablet_replica.cc:330] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a: stopping tablet replica
I20260618 22:42:35.040189 22526 raft_consensus.cc:2229] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:42:35.040743 22526 raft_consensus.cc:2258] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:42:35.041569 22527 tablet_replica.cc:330] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39: stopping tablet replica
I20260618 22:42:35.041709 22525 tablet_replica.cc:330] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df: stopping tablet replica
I20260618 22:42:35.042160 22527 raft_consensus.cc:2229] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:42:35.042893 22527 raft_consensus.cc:2258] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:42:35.043525 22525 raft_consensus.cc:2229] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:42:35.044623 22526 ts_tablet_manager.cc:1905] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:42:35.045185 22527 ts_tablet_manager.cc:1905] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:42:35.046876 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:35.029374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44734 (local address 127.5.213.193:35929)
0618 22:42:35.029540 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:35.029546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:35.029565 (+ 19us) server_negotiation.cc:407] Connection header received
0618 22:42:35.029662 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:35.029668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:35.029748 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:35.029861 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:35.031685 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:35.033061 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:35.035140 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:35.035823 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:35.035956 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:35.039801 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:35.039816 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:35.039835 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:35.039878 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:35.043716 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:35.044574 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:35.044583 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:35.044586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:35.044754 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:35.045148 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:35.045152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:35.045156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:35.045664 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:35.045950 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:35.046269 (+ 319us) server_negotiation.cc:299] Negotiation successful
0618 22:42:35.046494 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 22:42:35.047758 22218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:42:35.050777 22525 raft_consensus.cc:2258] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:42:35.052980 22527 ts_tablet_manager.cc:1918] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:42:35.053109 22525 ts_tablet_manager.cc:1905] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:42:35.053251 22527 log.cc:1196] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/0ac6e9d3925949e4ab31775927bd2104
I20260618 22:42:35.053905 22527 ts_tablet_manager.cc:1939] T 0ac6e9d3925949e4ab31775927bd2104 P 20848997b00049b8a471edcf3a537b39: Deleting consensus metadata
I20260618 22:42:35.056244 22025 catalog_manager.cc:4841] TS 20848997b00049b8a471edcf3a537b39 (127.5.213.195:34669): tablet 0ac6e9d3925949e4ab31775927bd2104 (table test-table [id=b53b25c8204c45ce93e1abb40a73215d]) successfully deleted
I20260618 22:42:35.061496 22525 ts_tablet_manager.cc:1918] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:42:35.061690 22525 log.cc:1196] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/0ac6e9d3925949e4ab31775927bd2104
I20260618 22:42:35.062280 22525 ts_tablet_manager.cc:1939] T 0ac6e9d3925949e4ab31775927bd2104 P 09b731a378054e73831d2313a823a2df: Deleting consensus metadata
I20260618 22:42:35.064298 22025 catalog_manager.cc:4841] TS 09b731a378054e73831d2313a823a2df (127.5.213.194:40135): tablet 0ac6e9d3925949e4ab31775927bd2104 (table test-table [id=b53b25c8204c45ce93e1abb40a73215d]) successfully deleted
I20260618 22:42:35.144222 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 22075
W20260618 22:42:35.158218 22023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.213.193:35929 (error 108)
I20260618 22:42:35.158969 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 22227
I20260618 22:42:35.160408 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:35.160005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35929 (local address 127.0.0.1:44742)
0618 22:42:35.160180 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:35.160264 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260618 22:42:35.160624 22023 catalog_manager.cc:4584] TS 573d81b018874aa3a49e60a70914643a (127.5.213.193:35929): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ac6e9d3925949e4ab31775927bd2104: Network error: Client connection negotiation failed: client connection to 127.5.213.193:35929: connect: Connection refused (error 111)
I20260618 22:42:35.174532 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:35.159741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.5.213.194:40135)
0618 22:42:35.159940 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:35.159946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:35.160550 (+ 604us) server_negotiation.cc:407] Connection header received
0618 22:42:35.160628 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:35.160634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:35.160715 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:35.160815 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:35.162530 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:35.163380 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:35.165198 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:35.165591 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:35.165704 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:35.168917 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:35.168933 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:35.168948 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:35.168995 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:35.171805 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:35.172662 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:35.172670 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:35.172672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:35.172829 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:35.173189 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:35.173193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:35.173195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:35.173559 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:35.173806 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:35.174075 (+ 269us) server_negotiation.cc:299] Negotiation successful
0618 22:42:35.174263 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260618 22:42:35.175376 22350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:42:35.214329 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:35.213781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35929 (local address 127.0.0.1:44754)
0618 22:42:35.214016 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:35.214147 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260618 22:42:35.215488 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:35.215070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35929 (local address 127.0.0.1:44770)
0618 22:42:35.215233 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:35.215338 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260618 22:42:35.264061 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 22227
I20260618 22:42:35.276202 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:35.275752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35929 (local address 127.0.0.1:44772)
0618 22:42:35.275960 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:35.276044 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260618 22:42:35.277243 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:35.276907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35929 (local address 127.0.0.1:44774)
0618 22:42:35.277053 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:35.277130 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260618 22:42:35.284283 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 22358
I20260618 22:42:35.300499 22501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:35.285138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37626 (local address 127.5.213.195:34669)
0618 22:42:35.285342 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:35.285348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:35.285370 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:42:35.285458 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:35.285464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:35.285550 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:35.285744 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:35.287178 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:35.288415 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:35.289762 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:35.290402 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:35.290534 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:35.293031 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:35.293045 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:35.293070 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:35.293116 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:35.297141 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:35.297877 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:35.297887 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:35.297891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:35.298072 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:35.298439 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:35.298447 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:35.298451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:35.299209 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:35.299530 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:35.299823 (+ 293us) server_negotiation.cc:299] Negotiation successful
0618 22:42:35.300077 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260618 22:42:35.301298 22481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:42:35.357613 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:35.357110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35929 (local address 127.0.0.1:44778)
0618 22:42:35.357363 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:35.357462 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260618 22:42:35.358740 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:35.358290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35929 (local address 127.0.0.1:44782)
0618 22:42:35.358486 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:35.358572 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260618 22:42:35.421839 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 22358
I20260618 22:42:35.441931 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 22005
I20260618 22:42:35.443058 22068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:42:35.554967 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:35.554308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35929 (local address 127.0.0.1:44784)
0618 22:42:35.554519 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:35.554672 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260618 22:42:35.555219 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 22005
I20260618 22:42:55.573220 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:45069
--webserver_interface=127.5.213.254
--webserver_port=36885
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:45069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:42:55.889990 22533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:42:55.890278 22533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:42:55.890420 22533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:42:55.897886 22533 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260618 22:42:55.898005 22533 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260618 22:42:55.898051 22533 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260618 22:42:55.898131 22533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:42:55.898204 22533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:42:55.898295 22533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:42:55.898351 22533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:42:55.898403 22533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:42:55.908720 22533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:45069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:45069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=36885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:42:55.910092 22533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:42:55.911542 22533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:42:55.923446 22539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:55.923415 22541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:55.923543 22538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:42:55.924036 22533 server_base.cc:1034] running on GCE node
I20260618 22:42:55.924813 22533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:42:55.926177 22533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:42:55.927409 22533 hybrid_clock.cc:648] HybridClock initialized: now 1781822575927348 us; error 36 us; skew 500 ppm
I20260618 22:42:55.927735 22533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:42:55 dist-test-slave-lnj3 krb5kdc[21455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822575, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:42:55.937204 22533 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:42:55.938813 22533 webserver.cc:460] Webserver started at http://127.5.213.254:36885/ using document root <none> and password file <none>
I20260618 22:42:55.939499 22533 fs_manager.cc:359] Metadata directory not provided
I20260618 22:42:55.939605 22533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:42:55.945214 22533 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260618 22:42:55.948076 22548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:55.949339 22533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260618 22:42:55.949486 22533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "9c06746ac2c24833879dde032c0ac934"
format_stamp: "Formatted at 2026-06-18 22:42:10 on dist-test-slave-lnj3"
server_key: "4f0ec6b4daa8cfce989761822a24395e"
server_key_iv: "d6dbb391f4db6184aee08c9ccf6fbc3e"
server_key_version: "encryptionkey@0"
I20260618 22:42:55.950397 22533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:42:55.966008 22533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:42:55.969879 22533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:42:55.970126 22533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:42:55.990870 22533 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:45069
I20260618 22:42:55.990893 22599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:45069 every 8 connection(s)
I20260618 22:42:55.992344 22533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:42:55.995092 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 22533
I20260618 22:42:55.995849 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:35929
--local_ip_for_outbound_sockets=127.5.213.193
--tserver_master_addrs=127.5.213.254:45069
--webserver_port=37697
--webserver_interface=127.5.213.193
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:42:55.999183 22600 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:42:56.002761 22600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934: Bootstrap starting.
I20260618 22:42:56.022475 22600 log.cc:824] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934: Log is configured to *not* fsync() on all Append() calls
I20260618 22:42:56.038215 22600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:42:56.038754 22600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934: Bootstrap complete.
I20260618 22:42:56.047549 22600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c06746ac2c24833879dde032c0ac934" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 45069 } }
I20260618 22:42:56.048413 22600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c06746ac2c24833879dde032c0ac934, State: Initialized, Role: FOLLOWER
I20260618 22:42:56.049014 22600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c06746ac2c24833879dde032c0ac934" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 45069 } }
I20260618 22:42:56.049247 22600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:42:56.049350 22600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:42:56.049496 22600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 2 FOLLOWER]: Advancing to term 3
I20260618 22:42:56.053227 22600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c06746ac2c24833879dde032c0ac934" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 45069 } }
I20260618 22:42:56.053675 22600 leader_election.cc:302] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c06746ac2c24833879dde032c0ac934; no voters:
I20260618 22:42:56.054287 22600 leader_election.cc:288] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260618 22:42:56.054514 22604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 3 FOLLOWER]: Leader election won for term 3
I20260618 22:42:56.055557 22604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [term 3 LEADER]: Becoming Leader. State: Replica: 9c06746ac2c24833879dde032c0ac934, State: Running, Role: LEADER
I20260618 22:42:56.056140 22604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c06746ac2c24833879dde032c0ac934" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 45069 } }
I20260618 22:42:56.056310 22600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:42:56.058470 22605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9c06746ac2c24833879dde032c0ac934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c06746ac2c24833879dde032c0ac934" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 45069 } } }
I20260618 22:42:56.058846 22605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [sys.catalog]: This master's current role is: LEADER
I20260618 22:42:56.063858 22606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c06746ac2c24833879dde032c0ac934. Latest consensus state: current_term: 3 leader_uuid: "9c06746ac2c24833879dde032c0ac934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c06746ac2c24833879dde032c0ac934" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 45069 } } }
I20260618 22:42:56.064146 22606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934 [sys.catalog]: This master's current role is: LEADER
I20260618 22:42:56.066355 22610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:42:56.075299 22610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:42:56.076792 22610 catalog_manager.cc:1259] Loaded cluster ID: 7b998e51a5fb427491a1a0ade82e0a8c
I20260618 22:42:56.076948 22610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:42:56.080960 22610 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:42:56.082513 22610 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934: Loaded TSK: 0
I20260618 22:42:56.082747 22610 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9c06746ac2c24833879dde032c0ac934: Loaded TSK: 1
F20260618 22:42:56.083870 22610 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f619e3439ad google::LogMessage::Fail() at ??:0
@ 0x7f619e34688a google::LogMessage::SendToLog() at ??:0
@ 0x7f619e3434de google::LogMessage::Flush() at ??:0
@ 0x7f619e344a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f61a20a77d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f61a209673e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f61b3d721c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f61b3d71783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f61b3dfd2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f61b3dfd0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f61b3e475e9 std::function<>::operator()() at ??:0
@ 0x7f61b3d75cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f61b3d73c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f61b3dfb610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f61b3dfb439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f61a0f8f345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f61a0f9291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f61a0f92799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f61a0f6deaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f61a42476db start_thread at ??:0
@ 0x7f619d1dd61f clone at ??:0
W20260618 22:42:56.366941 22602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:42:56.367211 22602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:42:56.367334 22602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:42:56.374444 22602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:42:56.374545 22602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:42:56.374742 22602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:42:56.384156 22602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:35929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=37697
--tserver_master_addrs=127.5.213.254:45069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:42:56.385324 22602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:42:56.386581 22602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:42:56.399449 22648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:56.399488 22647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:56.399511 22650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:42:56.400756 22602 server_base.cc:1034] running on GCE node
I20260618 22:42:56.401300 22602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:42:56.402148 22602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:42:56.403360 22602 hybrid_clock.cc:648] HybridClock initialized: now 1781822576403313 us; error 33 us; skew 500 ppm
I20260618 22:42:56.403667 22602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:42:56 dist-test-slave-lnj3 krb5kdc[21455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822576, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:42:56.406840 22602 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:42:56.408195 22602 webserver.cc:460] Webserver started at http://127.5.213.193:37697/ using document root <none> and password file <none>
I20260618 22:42:56.408842 22602 fs_manager.cc:359] Metadata directory not provided
I20260618 22:42:56.408938 22602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:42:56.413976 22602 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260618 22:42:56.416793 22657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:56.418113 22602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260618 22:42:56.418244 22602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "573d81b018874aa3a49e60a70914643a"
format_stamp: "Formatted at 2026-06-18 22:42:11 on dist-test-slave-lnj3"
server_key: "d970bca08826606cebdc5be6c8c3d04e"
server_key_iv: "77c5d1e46d0f92258e064582dd2fc8ca"
server_key_version: "encryptionkey@0"
I20260618 22:42:56.419212 22602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:42:56.438227 22602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:42:56.442036 22602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:42:56.442262 22602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:42:56.443194 22602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:42:56.444834 22664 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260618 22:42:56.450192 22602 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 22:42:56.450299 22602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260618 22:42:56.450389 22602 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 22:42:56.453555 22602 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 22:42:56.453625 22602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260618 22:42:56.454010 22664 tablet_bootstrap.cc:492] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a: Bootstrap starting.
I20260618 22:42:56.461987 22664 log.cc:824] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:42:56.477308 22664 tablet_bootstrap.cc:492] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:42:56.478058 22664 tablet_bootstrap.cc:492] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a: Bootstrap complete.
I20260618 22:42:56.478595 22664 ts_tablet_manager.cc:1397] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a: Time spent bootstrapping tablet: real 0.025s user 0.019s sys 0.004s
I20260618 22:42:56.488049 22602 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:35929
I20260618 22:42:56.488101 22771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:35929 every 8 connection(s)
I20260618 22:42:56.487942 22664 raft_consensus.cc:348] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } }
I20260618 22:42:56.488997 22664 raft_consensus.cc:729] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573d81b018874aa3a49e60a70914643a, State: Initialized, Role: FOLLOWER
I20260618 22:42:56.489377 22602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:42:56.489655 22664 consensus_queue.cc:260] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } }
I20260618 22:42:56.491462 22664 ts_tablet_manager.cc:1428] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260618 22:42:56.494177 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:56.492389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.193:34349)
0618 22:42:56.493102 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:56.493329 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":306,"threads_started":1}
W20260618 22:42:56.494831 22772 heartbeater.cc:644] Failed to heartbeat to 127.5.213.254:45069 (0 consecutive failures): Network error: Failed to ping master at 127.5.213.254:45069: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
I20260618 22:42:56.495648 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:56.495300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.193:42577)
0618 22:42:56.495452 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:56.495522 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260618 22:42:56.496464 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:56.496155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.193:44675)
0618 22:42:56.496269 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:56.496353 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260618 22:42:56.496671 22772 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 22:42:56.498229 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 22602
I20260618 22:42:56.498899 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:40135
--local_ip_for_outbound_sockets=127.5.213.194
--tserver_master_addrs=127.5.213.254:45069
--webserver_port=38561
--webserver_interface=127.5.213.194
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:42:56.811451 22776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:42:56.811789 22776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:42:56.811888 22776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:42:56.819095 22776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:42:56.819204 22776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:42:56.819322 22776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:42:56.829147 22776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:40135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=38561
--tserver_master_addrs=127.5.213.254:45069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:42:56.830346 22776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:42:56.831632 22776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:42:56.843961 22782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:56.843910 22781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:56.844550 22784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:42:56.844853 22776 server_base.cc:1034] running on GCE node
I20260618 22:42:56.845402 22776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:42:56.846235 22776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:42:56.847437 22776 hybrid_clock.cc:648] HybridClock initialized: now 1781822576847390 us; error 54 us; skew 500 ppm
I20260618 22:42:56.847759 22776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:42:56 dist-test-slave-lnj3 krb5kdc[21455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822576, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:42:56.850801 22776 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:42:56.852159 22776 webserver.cc:460] Webserver started at http://127.5.213.194:38561/ using document root <none> and password file <none>
I20260618 22:42:56.852811 22776 fs_manager.cc:359] Metadata directory not provided
I20260618 22:42:56.852924 22776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:42:56.858104 22776 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:42:56.860754 22791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:56.861994 22776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:42:56.862141 22776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "09b731a378054e73831d2313a823a2df"
format_stamp: "Formatted at 2026-06-18 22:42:11 on dist-test-slave-lnj3"
server_key: "ac955d938b10fc03cf0ad3f4c4b84024"
server_key_iv: "6a497b162b618903fef4ea74e81a09fe"
server_key_version: "encryptionkey@0"
I20260618 22:42:56.863142 22776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:42:56.882476 22776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:42:56.886504 22776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:42:56.886785 22776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:42:56.887741 22776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:42:56.889026 22776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:42:56.889123 22776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:56.889230 22776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:42:56.889297 22776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:56.919291 22776 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:40135
I20260618 22:42:56.919355 22903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:40135 every 8 connection(s)
I20260618 22:42:56.920701 22776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:42:56.925462 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:56.923651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.194:46649)
0618 22:42:56.924368 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:56.924594 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":265,"threads_started":1}
W20260618 22:42:56.926122 22904 heartbeater.cc:644] Failed to heartbeat to 127.5.213.254:45069 (0 consecutive failures): Network error: Failed to ping master at 127.5.213.254:45069: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
I20260618 22:42:56.926935 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:56.926578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.194:50217)
0618 22:42:56.926749 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:56.926837 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260618 22:42:56.927668 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:56.927372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.194:41445)
0618 22:42:56.927483 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:56.927555 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260618 22:42:56.927873 22904 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 22:42:56.930039 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 22776
I20260618 22:42:56.930596 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:34669
--local_ip_for_outbound_sockets=127.5.213.195
--tserver_master_addrs=127.5.213.254:45069
--webserver_port=43319
--webserver_interface=127.5.213.195
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:42:57.244203 22907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:42:57.244531 22907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:42:57.244691 22907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:42:57.254300 22907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:42:57.254406 22907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:42:57.254554 22907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:42:57.264549 22907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:34669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=43319
--tserver_master_addrs=127.5.213.254:45069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:42:57.265738 22907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:42:57.267149 22907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:42:57.279373 22913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:57.279414 22912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:42:57.279332 22915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:42:57.280211 22907 server_base.cc:1034] running on GCE node
I20260618 22:42:57.280776 22907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:42:57.281685 22907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:42:57.282869 22907 hybrid_clock.cc:648] HybridClock initialized: now 1781822577282827 us; error 39 us; skew 500 ppm
I20260618 22:42:57.283159 22907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:42:57 dist-test-slave-lnj3 krb5kdc[21455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822577, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:42:57.286092 22907 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:42:57.287495 22907 webserver.cc:460] Webserver started at http://127.5.213.195:43319/ using document root <none> and password file <none>
I20260618 22:42:57.288160 22907 fs_manager.cc:359] Metadata directory not provided
I20260618 22:42:57.288262 22907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:42:57.293318 22907 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:42:57.295861 22922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:57.297108 22907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:42:57.297235 22907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "20848997b00049b8a471edcf3a537b39"
format_stamp: "Formatted at 2026-06-18 22:42:12 on dist-test-slave-lnj3"
server_key: "553c035740e5989fe68d908038d4840c"
server_key_iv: "73d7b5f1aec13a420a56553bc2c84df6"
server_key_version: "encryptionkey@0"
I20260618 22:42:57.298125 22907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:42:57.310797 22907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:42:57.314610 22907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:42:57.314867 22907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:42:57.315733 22907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:42:57.317040 22907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:42:57.317131 22907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:57.317207 22907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:42:57.317262 22907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:42:57.347460 22907 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:34669
I20260618 22:42:57.347512 23034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:34669 every 8 connection(s)
I20260618 22:42:57.348848 22907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:42:57.350327 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 22907
I20260618 22:42:57.354238 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:57.352463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.195:45703)
0618 22:42:57.353157 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:57.353383 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":272,"threads_started":1}
W20260618 22:42:57.354822 23035 heartbeater.cc:644] Failed to heartbeat to 127.5.213.254:45069 (0 consecutive failures): Network error: Failed to ping master at 127.5.213.254:45069: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
I20260618 22:42:57.355540 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:57.355254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.195:60027)
0618 22:42:57.355378 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:57.355441 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260618 22:42:57.356328 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:57.356018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.195:41007)
0618 22:42:57.356135 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:57.356208 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260618 22:42:57.356550 23035 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 22:42:57.497838 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:57.497135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.193:41503)
0618 22:42:57.497529 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:57.497604 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":179,"threads_started":1}
I20260618 22:42:57.838913 23040 raft_consensus.cc:482] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:42:57.839202 23040 raft_consensus.cc:504] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } }
I20260618 22:42:57.840716 23040 leader_election.cc:288] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09b731a378054e73831d2313a823a2df (127.5.213.194:40135), 20848997b00049b8a471edcf3a537b39 (127.5.213.195:34669)
Jun 18 22:42:57 dist-test-slave-lnj3 krb5kdc[21455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822576, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.195@KRBTEST.COM
Jun 18 22:42:57 dist-test-slave-lnj3 krb5kdc[21455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822576, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.194@KRBTEST.COM
I20260618 22:42:57.861560 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:57.840971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:40135 (local address 127.5.213.193:37959)
0618 22:42:57.841150 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:57.841186 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0618 22:42:57.841748 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:42:57.844974 (+ 3226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:42:57.844984 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:42:57.845665 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:42:57.846423 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:57.846440 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:57.847991 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:57.847996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:57.849469 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:57.849480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:57.850467 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:57.850477 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:57.850686 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:57.851349 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:42:57.851370 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:42:57.853966 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:42:57.857714 (+ 3748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:42:57.857724 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:42:57.857740 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:42:57.858468 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:42:57.858958 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:42:57.858962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:42:57.858965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:42:57.859212 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:42:57.860074 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:42:57.860080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:42:57.860367 (+ 287us) client_negotiation.cc:769] Sending connection context
0618 22:42:57.860606 (+ 239us) client_negotiation.cc:240] Negotiation successful
0618 22:42:57.861126 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":57,"spinlock_wait_cycles":23936}
I20260618 22:42:57.861627 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:57.841210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:37959 (local address 127.5.213.194:40135)
0618 22:42:57.842605 (+ 1395us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:57.842621 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:57.842682 (+ 61us) server_negotiation.cc:407] Connection header received
0618 22:42:57.844175 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:57.844183 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:57.844542 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:57.844835 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:57.846603 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:57.847795 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:57.849602 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:57.850194 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:57.850357 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:57.854200 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:57.854209 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:57.854224 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:57.854270 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:57.857560 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:57.858672 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:57.858681 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:57.858688 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:57.858860 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:57.859321 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:57.859326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:57.859329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:57.859941 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:57.860077 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:57.860858 (+ 781us) server_negotiation.cc:299] Negotiation successful
0618 22:42:57.861262 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1194,"thread_start_us":1059,"threads_started":1}
I20260618 22:42:57.861954 23043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:57.841718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:58147 (local address 127.5.213.195:34669)
0618 22:42:57.843166 (+ 1448us) server_negotiation.cc:206] Beginning negotiation
0618 22:42:57.843176 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:42:57.843207 (+ 31us) server_negotiation.cc:407] Connection header received
0618 22:42:57.844571 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:42:57.844579 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:42:57.844884 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:42:57.845143 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:42:57.846599 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:57.847777 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:57.849604 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:42:57.850213 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:42:57.850372 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:57.853963 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:42:57.853970 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:42:57.853983 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:42:57.854025 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:42:57.857560 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:57.858602 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:57.858610 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:57.858618 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:57.858841 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:42:57.860128 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:42:57.860131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:42:57.860133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:42:57.860565 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:42:57.860731 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0618 22:42:57.861216 (+ 485us) server_negotiation.cc:299] Negotiation successful
0618 22:42:57.861622 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1271,"thread_start_us":1132,"threads_started":1}
I20260618 22:42:57.861653 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:57.841384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:34669 (local address 127.5.213.193:58147)
0618 22:42:57.842922 (+ 1538us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:57.842939 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:42:57.843049 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:42:57.845120 (+ 2071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:42:57.845124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:42:57.845700 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:42:57.846420 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:57.846439 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:57.847978 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:57.847982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:57.849471 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:42:57.849482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:42:57.850391 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:42:57.850398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:42:57.850667 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:42:57.851411 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:42:57.851428 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:42:57.853696 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:42:57.857714 (+ 4018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:42:57.857724 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:42:57.857740 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:42:57.858468 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:42:57.858948 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:42:57.858953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:42:57.858956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:42:57.859212 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:42:57.860707 (+ 1495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:42:57.860712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:42:57.860843 (+ 131us) client_negotiation.cc:769] Sending connection context
0618 22:42:57.860973 (+ 130us) client_negotiation.cc:240] Negotiation successful
0618 22:42:57.861263 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1402,"mutex_wait_us":131,"spinlock_wait_cycles":25088,"thread_start_us":1243,"threads_started":1}
W20260618 22:42:57.864913 22661 leader_election.cc:341] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09b731a378054e73831d2313a823a2df (127.5.213.194:40135): Not found: Tablet not found: 0ac6e9d3925949e4ab31775927bd2104
W20260618 22:42:57.865195 22661 leader_election.cc:341] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20848997b00049b8a471edcf3a537b39 (127.5.213.195:34669): Not found: Tablet not found: 0ac6e9d3925949e4ab31775927bd2104
I20260618 22:42:57.865300 22661 leader_election.cc:302] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 573d81b018874aa3a49e60a70914643a; no voters: 09b731a378054e73831d2313a823a2df, 20848997b00049b8a471edcf3a537b39
I20260618 22:42:57.865635 23040 raft_consensus.cc:2726] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:42:57.929203 23044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:57.928354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.194:50873)
0618 22:42:57.928850 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:57.928983 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":268,"threads_started":1}
I20260618 22:42:58.357968 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:58.357131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.195:54913)
0618 22:42:58.357600 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:58.357753 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":226,"threads_started":1}
I20260618 22:42:58.499326 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:58.498668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.193:43849)
0618 22:42:58.499081 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:58.499172 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":205,"threads_started":1}
I20260618 22:42:58.930734 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:58.930039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.194:41001)
0618 22:42:58.930442 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:58.930525 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":204,"threads_started":1}
I20260618 22:42:59.359503 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:59.358840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.195:46903)
0618 22:42:59.359266 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:59.359347 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":223,"threads_started":1}
I20260618 22:42:59.500906 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:59.500147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.193:55039)
0618 22:42:59.500629 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:59.500714 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":239,"threads_started":1}
I20260618 22:42:59.578109 23050 raft_consensus.cc:482] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:42:59.578330 23050 raft_consensus.cc:504] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } }
I20260618 22:42:59.579020 23050 leader_election.cc:288] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09b731a378054e73831d2313a823a2df (127.5.213.194:40135), 20848997b00049b8a471edcf3a537b39 (127.5.213.195:34669)
W20260618 22:42:59.580112 22661 leader_election.cc:341] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09b731a378054e73831d2313a823a2df (127.5.213.194:40135): Not found: Tablet not found: 0ac6e9d3925949e4ab31775927bd2104
W20260618 22:42:59.580356 22661 leader_election.cc:341] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20848997b00049b8a471edcf3a537b39 (127.5.213.195:34669): Not found: Tablet not found: 0ac6e9d3925949e4ab31775927bd2104
I20260618 22:42:59.580444 22661 leader_election.cc:302] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 573d81b018874aa3a49e60a70914643a; no voters: 09b731a378054e73831d2313a823a2df, 20848997b00049b8a471edcf3a537b39
I20260618 22:42:59.580708 23050 raft_consensus.cc:2726] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:42:59.932132 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:42:59.931472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.194:37503)
0618 22:42:59.931897 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0618 22:42:59.931977 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":224,"threads_started":1}
I20260618 22:43:00.361104 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:00.360359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.195:47815)
0618 22:43:00.360850 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:00.360943 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":262,"threads_started":1}
I20260618 22:43:00.502468 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:00.501803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.193:42783)
0618 22:43:00.502220 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:00.502306 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":210,"threads_started":1}
I20260618 22:43:00.933894 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:00.933026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.194:50531)
0618 22:43:00.933581 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:00.933679 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":340,"threads_started":1}
I20260618 22:43:01.362807 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:01.362021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.195:47423)
0618 22:43:01.362478 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:01.362559 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":226,"threads_started":1}
I20260618 22:43:01.504235 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:01.503386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.193:48897)
0618 22:43:01.503908 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:01.504000 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":237,"threads_started":1}
I20260618 22:43:01.935766 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:01.935001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.194:58477)
0618 22:43:01.935476 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:01.935560 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":257,"threads_started":1}
I20260618 22:43:01.958783 23058 raft_consensus.cc:482] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:43:01.958999 23058 raft_consensus.cc:504] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } }
I20260618 22:43:01.959627 23058 leader_election.cc:288] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09b731a378054e73831d2313a823a2df (127.5.213.194:40135), 20848997b00049b8a471edcf3a537b39 (127.5.213.195:34669)
W20260618 22:43:01.960731 22661 leader_election.cc:341] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20848997b00049b8a471edcf3a537b39 (127.5.213.195:34669): Not found: Tablet not found: 0ac6e9d3925949e4ab31775927bd2104
W20260618 22:43:01.960942 22661 leader_election.cc:341] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09b731a378054e73831d2313a823a2df (127.5.213.194:40135): Not found: Tablet not found: 0ac6e9d3925949e4ab31775927bd2104
I20260618 22:43:01.961054 22661 leader_election.cc:302] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 573d81b018874aa3a49e60a70914643a; no voters: 09b731a378054e73831d2313a823a2df, 20848997b00049b8a471edcf3a537b39
I20260618 22:43:01.961311 23058 raft_consensus.cc:2726] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:43:02.364578 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:02.363865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.195:47709)
0618 22:43:02.364332 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:02.364420 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":252,"threads_started":1}
I20260618 22:43:02.505918 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:02.505172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.193:33009)
0618 22:43:02.505647 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:02.505764 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":270,"threads_started":1}
I20260618 22:43:02.937520 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:02.936750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.194:48757)
0618 22:43:02.937252 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:02.937345 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":266,"threads_started":1}
I20260618 22:43:03.366406 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:03.365560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.195:58245)
0618 22:43:03.366067 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:03.366185 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":281,"threads_started":1}
I20260618 22:43:03.507786 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:03.506971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.193:37811)
0618 22:43:03.507464 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:03.507558 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":249,"threads_started":1}
I20260618 22:43:03.939232 23064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:03.938403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.194:54615)
0618 22:43:03.938956 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:03.939039 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":278,"threads_started":1}
I20260618 22:43:03.951491 23065 raft_consensus.cc:482] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:43:03.951750 23065 raft_consensus.cc:504] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } }
I20260618 22:43:03.952322 23065 leader_election.cc:288] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09b731a378054e73831d2313a823a2df (127.5.213.194:40135), 20848997b00049b8a471edcf3a537b39 (127.5.213.195:34669)
W20260618 22:43:03.953276 22661 leader_election.cc:341] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09b731a378054e73831d2313a823a2df (127.5.213.194:40135): Not found: Tablet not found: 0ac6e9d3925949e4ab31775927bd2104
W20260618 22:43:03.953483 22661 leader_election.cc:341] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20848997b00049b8a471edcf3a537b39 (127.5.213.195:34669): Not found: Tablet not found: 0ac6e9d3925949e4ab31775927bd2104
I20260618 22:43:03.953558 22661 leader_election.cc:302] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 573d81b018874aa3a49e60a70914643a; no voters: 09b731a378054e73831d2313a823a2df, 20848997b00049b8a471edcf3a537b39
I20260618 22:43:03.953770 23065 raft_consensus.cc:2726] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:43:04.368084 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:04.367314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.195:53815)
0618 22:43:04.367815 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:04.367900 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":231,"threads_started":1}
I20260618 22:43:04.509428 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:04.508711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.193:48731)
0618 22:43:04.509178 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:04.509269 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":232,"threads_started":1}
I20260618 22:43:04.941059 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:04.940294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.194:36339)
0618 22:43:04.940777 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:04.940869 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":230,"threads_started":1}
I20260618 22:43:05.369606 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:05.368924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.195:45963)
0618 22:43:05.369330 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:05.369408 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":211,"threads_started":1}
I20260618 22:43:05.511375 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:05.510524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.193:42363)
0618 22:43:05.511079 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:05.511186 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":299,"threads_started":1}
I20260618 22:43:05.942862 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:05.942059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.194:41233)
0618 22:43:05.942553 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:05.942687 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":229,"threads_started":1}
I20260618 22:43:06.371551 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:06.370487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.195:59025)
0618 22:43:06.371312 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:06.371398 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":547,"threads_started":1}
I20260618 22:43:06.513254 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:06.512499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.193:56781)
0618 22:43:06.512975 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:06.513075 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":251,"threads_started":1}
I20260618 22:43:06.944437 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:06.943761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.194:51207)
0618 22:43:06.944154 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:06.944233 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":204,"threads_started":1}
I20260618 22:43:07.373255 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:07.372438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.195:54913)
0618 22:43:07.372891 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:07.372998 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":253,"threads_started":1}
I20260618 22:43:07.515151 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:07.514310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.193:41785)
0618 22:43:07.514887 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:07.514980 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":244,"threads_started":1}
I20260618 22:43:07.946102 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:07.945351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.194:48595)
0618 22:43:07.945837 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:07.945936 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":231,"threads_started":1}
I20260618 22:43:08.374969 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:08.374180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.195:51569)
0618 22:43:08.374705 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:08.374810 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":226,"threads_started":1}
I20260618 22:43:08.516939 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:08.516108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.193:37493)
0618 22:43:08.516562 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:08.516653 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":228,"threads_started":1}
I20260618 22:43:08.947820 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:08.947015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.194:40937)
0618 22:43:08.947531 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:08.947631 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":280,"threads_started":1}
I20260618 22:43:09.376682 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:09.375946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.195:36919)
0618 22:43:09.376400 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:09.376484 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":227,"threads_started":1}
I20260618 22:43:09.518908 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:09.517971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.193:36637)
0618 22:43:09.518511 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:09.518702 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":298,"threads_started":1}
I20260618 22:43:09.949482 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:09.948782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.194:55371)
0618 22:43:09.949222 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:09.949318 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":213,"threads_started":1}
I20260618 22:43:10.182037 23084 raft_consensus.cc:482] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:43:10.182219 23084 raft_consensus.cc:504] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b731a378054e73831d2313a823a2df" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 40135 } } peers { permanent_uuid: "573d81b018874aa3a49e60a70914643a" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35929 } } peers { permanent_uuid: "20848997b00049b8a471edcf3a537b39" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34669 } }
I20260618 22:43:10.182915 23084 leader_election.cc:288] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09b731a378054e73831d2313a823a2df (127.5.213.194:40135), 20848997b00049b8a471edcf3a537b39 (127.5.213.195:34669)
W20260618 22:43:10.183775 22661 leader_election.cc:341] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09b731a378054e73831d2313a823a2df (127.5.213.194:40135): Not found: Tablet not found: 0ac6e9d3925949e4ab31775927bd2104
W20260618 22:43:10.183974 22661 leader_election.cc:341] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20848997b00049b8a471edcf3a537b39 (127.5.213.195:34669): Not found: Tablet not found: 0ac6e9d3925949e4ab31775927bd2104
I20260618 22:43:10.184077 22661 leader_election.cc:302] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 573d81b018874aa3a49e60a70914643a; no voters: 09b731a378054e73831d2313a823a2df, 20848997b00049b8a471edcf3a537b39
I20260618 22:43:10.184300 23084 raft_consensus.cc:2726] T 0ac6e9d3925949e4ab31775927bd2104 P 573d81b018874aa3a49e60a70914643a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:43:10.378242 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:10.377524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.195:44523)
0618 22:43:10.377980 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:10.378068 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":238,"threads_started":1}
I20260618 22:43:10.520854 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:10.519825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.193:50409)
0618 22:43:10.520444 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:10.520554 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":291,"threads_started":1}
I20260618 22:43:10.951278 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:10.950446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.194:47659)
0618 22:43:10.950982 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:10.951079 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":309,"threads_started":1}
I20260618 22:43:11.379863 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:11.379153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.195:60367)
0618 22:43:11.379580 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:11.379669 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":211,"threads_started":1}
I20260618 22:43:11.522850 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:11.522027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.193:42425)
0618 22:43:11.522505 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:11.522661 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":253,"threads_started":1}
I20260618 22:43:11.952988 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:11.952291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.194:47895)
0618 22:43:11.952711 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:11.952834 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":214,"threads_started":1}
I20260618 22:43:12.360013 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 22602
I20260618 22:43:12.378506 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:12.361195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48598 (local address 127.5.213.193:35929)
0618 22:43:12.361708 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:12.361715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:12.361743 (+ 28us) server_negotiation.cc:407] Connection header received
0618 22:43:12.361802 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:12.361808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:12.362020 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:12.362178 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:12.363712 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:12.364719 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:12.365938 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:12.366557 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:12.366716 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:12.373168 (+ 6452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:12.373180 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:12.373195 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:12.373230 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:12.375845 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:12.376539 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:12.376545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:12.376548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:12.376684 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:12.377014 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:12.377017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:12.377019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:12.377368 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:12.377671 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:12.377936 (+ 265us) server_negotiation.cc:299] Negotiation successful
0618 22:43:12.378196 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":230,"threads_started":1}
I20260618 22:43:12.379856 22767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:12.381871 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:12.381070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:45069 (local address 127.5.213.195:54697)
0618 22:43:12.381522 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:12.381701 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.254:45069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":234,"threads_started":1}
I20260618 22:43:12.469537 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 22602
I20260618 22:43:12.483934 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 22776
I20260618 22:43:12.496426 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:12.484674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53082 (local address 127.5.213.194:40135)
0618 22:43:12.485075 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:12.485080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:12.485096 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:43:12.485202 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:12.485207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:12.485268 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:12.485355 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:12.486545 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:12.487328 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:12.488617 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:12.489030 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:12.489127 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:12.491671 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:12.491678 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:12.491681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:12.491715 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:12.494242 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:12.494861 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:12.494864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:12.494867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:12.494986 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:12.495265 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:12.495268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:12.495270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:12.495519 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:12.495721 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:12.495894 (+ 173us) server_negotiation.cc:299] Negotiation successful
0618 22:43:12.496053 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":206,"threads_started":1}
I20260618 22:43:12.497107 22899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:12.579066 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 22776
I20260618 22:43:12.592991 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 22907
I20260618 22:43:12.606395 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:12.593795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60524 (local address 127.5.213.195:34669)
0618 22:43:12.594321 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:12.594326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:12.594345 (+ 19us) server_negotiation.cc:407] Connection header received
0618 22:43:12.594399 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:12.594403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:12.594471 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:12.594561 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:12.595939 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:12.596738 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:12.598030 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:12.598445 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:12.598534 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:12.601312 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:12.601317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:12.601320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:12.601354 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:12.604081 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:12.604704 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:12.604708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:12.604710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:12.604880 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:12.605152 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:12.605155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:12.605157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:12.605398 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:12.605595 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:12.605871 (+ 276us) server_negotiation.cc:299] Negotiation successful
0618 22:43:12.606031 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":257,"threads_started":1}
I20260618 22:43:12.607302 23030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:12.690578 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 22907
2026-06-18T22:43:12Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63778 ms)
[----------] 30 tests from SecurityITest (284821 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:43:12 dist-test-slave-lnj3 krb5kdc[23100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:43:12 dist-test-slave-lnj3 krb5kdc[23100](info): set up 2 sockets
Jun 18 22:43:12 dist-test-slave-lnj3 krb5kdc[23100](info): commencing operation
krb5kdc: starting...
W20260618 22:43:13.714666 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:43:13 dist-test-slave-lnj3 krb5kdc[23100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:43:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:43:13.938356 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:46213
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:39215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:46213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:14.255256 23113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:14.255540 23113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:14.255674 23113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:14.262754 23113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:43:14.262848 23113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:14.262925 23113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:14.263003 23113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:43:14.263062 23113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:43:14.272869 23113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:39215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:46213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:46213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:14.274129 23113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:14.275450 23113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:14.286883 23121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:14.286945 23119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:14.287245 23113 server_base.cc:1034] running on GCE node
W20260618 22:43:14.286948 23118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:14.288266 23113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:14.289686 23113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:14.290887 23113 hybrid_clock.cc:648] HybridClock initialized: now 1781822594290834 us; error 43 us; skew 500 ppm
I20260618 22:43:14.291213 23113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:14 dist-test-slave-lnj3 krb5kdc[23100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822594, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:14.294210 23113 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:43:14.295694 23113 webserver.cc:460] Webserver started at http://127.5.213.254:42535/ using document root <none> and password file <none>
I20260618 22:43:14.296365 23113 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:14.296481 23113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:14.296766 23113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:14.299772 23113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "c135fe2858b641d799364cd20afe95a2"
format_stamp: "Formatted at 2026-06-18 22:43:14 on dist-test-slave-lnj3"
server_key: "d017414c698a8155b22e6e65c804c4ff"
server_key_iv: "7fe2bb6cd3c858733b46618416a8526a"
server_key_version: "encryptionkey@0"
I20260618 22:43:14.300475 23113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "c135fe2858b641d799364cd20afe95a2"
format_stamp: "Formatted at 2026-06-18 22:43:14 on dist-test-slave-lnj3"
server_key: "d017414c698a8155b22e6e65c804c4ff"
server_key_iv: "7fe2bb6cd3c858733b46618416a8526a"
server_key_version: "encryptionkey@0"
I20260618 22:43:14.305394 23113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260618 22:43:14.309059 23128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:14.310454 23113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260618 22:43:14.310618 23113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "c135fe2858b641d799364cd20afe95a2"
format_stamp: "Formatted at 2026-06-18 22:43:14 on dist-test-slave-lnj3"
server_key: "d017414c698a8155b22e6e65c804c4ff"
server_key_iv: "7fe2bb6cd3c858733b46618416a8526a"
server_key_version: "encryptionkey@0"
I20260618 22:43:14.310839 23113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:14.336565 23113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:14.340718 23113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:14.340972 23113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:14.358361 23113 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:46213
I20260618 22:43:14.358368 23179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:46213 every 8 connection(s)
I20260618 22:43:14.359732 23113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:43:14.363456 23180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:14.369380 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 23113
I20260618 22:43:14.369561 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:43:14.370191 5975 external_mini_cluster.cc:1428] Setting key fa3d6b6643a0ab7f9804444fe22eeed5
I20260618 22:43:14.373648 23180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2: Bootstrap starting.
I20260618 22:43:14.377214 23180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:14.378247 23180 log.cc:824] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:14.381067 23180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2: No bootstrap required, opened a new log
Jun 18 22:43:14 dist-test-slave-lnj3 krb5kdc[23100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:43:14.390168 23180 raft_consensus.cc:348] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c135fe2858b641d799364cd20afe95a2" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 46213 } }
I20260618 22:43:14.390652 23180 raft_consensus.cc:374] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:14.390765 23180 raft_consensus.cc:729] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c135fe2858b641d799364cd20afe95a2, State: Initialized, Role: FOLLOWER
I20260618 22:43:14.391342 23180 consensus_queue.cc:260] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c135fe2858b641d799364cd20afe95a2" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 46213 } }
I20260618 22:43:14.391623 23180 raft_consensus.cc:388] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:43:14.391783 23180 raft_consensus.cc:482] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:43:14.391954 23180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:14.394464 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:14.372083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45716 (local address 127.5.213.254:46213)
0618 22:43:14.372859 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:14.372870 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:14.372909 (+ 39us) server_negotiation.cc:407] Connection header received
0618 22:43:14.374646 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:14.374676 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:14.375079 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:14.375402 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:14.377178 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:14.378595 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:14.380254 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:14.380988 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:14.381176 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:14.385923 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:14.385931 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:14.385945 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:14.385985 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:14.389183 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:14.389933 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:14.389939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:14.389946 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:14.390107 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:14.390517 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:14.390521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:14.390523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:14.391242 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:14.391465 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:14.392584 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0618 22:43:14.393058 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":366,"threads_started":1}
I20260618 22:43:14.395216 23180 raft_consensus.cc:504] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c135fe2858b641d799364cd20afe95a2" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 46213 } }
I20260618 22:43:14.395828 23180 leader_election.cc:302] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c135fe2858b641d799364cd20afe95a2; no voters:
I20260618 22:43:14.397107 23180 leader_election.cc:288] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:43:14.397260 23185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:14.398348 23185 raft_consensus.cc:686] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2 [term 1 LEADER]: Becoming Leader. State: Replica: c135fe2858b641d799364cd20afe95a2, State: Running, Role: LEADER
I20260618 22:43:14.398932 23185 consensus_queue.cc:237] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c135fe2858b641d799364cd20afe95a2" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 46213 } }
I20260618 22:43:14.399262 23180 sys_catalog.cc:564] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:43:14.401650 23186 sys_catalog.cc:455] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c135fe2858b641d799364cd20afe95a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c135fe2858b641d799364cd20afe95a2" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 46213 } } }
I20260618 22:43:14.402046 23186 sys_catalog.cc:458] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:14.401640 23187 sys_catalog.cc:455] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c135fe2858b641d799364cd20afe95a2. Latest consensus state: current_term: 1 leader_uuid: "c135fe2858b641d799364cd20afe95a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c135fe2858b641d799364cd20afe95a2" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 46213 } } }
I20260618 22:43:14.402523 23187 sys_catalog.cc:458] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:14.403013 23192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:43:14.408603 23192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:43:14.416630 23192 catalog_manager.cc:1347] Generated new cluster ID: 5f74ddb4f2b04dcc9b71ec8ea8d20ef2
I20260618 22:43:14.416733 23192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:43:14.425168 23192 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:43:14.426451 23192 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:43:14.437011 23192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2: Generated new TSK 0
I20260618 22:43:14.437867 23192 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
I20260618 22:43:14.492408 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:46213
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:39215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:14.810060 23206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:14.810333 23206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:14.810424 23206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:14.817900 23206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:14.818015 23206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:14.818137 23206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:43:14.827943 23206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:39215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:46213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:14.829142 23206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:14.830372 23206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:14.843068 23212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:14.843225 23211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:14.843603 23214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:14.843816 23206 server_base.cc:1034] running on GCE node
I20260618 22:43:14.844394 23206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:14.845278 23206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:14.846464 23206 hybrid_clock.cc:648] HybridClock initialized: now 1781822594846414 us; error 41 us; skew 500 ppm
I20260618 22:43:14.846779 23206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:14 dist-test-slave-lnj3 krb5kdc[23100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822594, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:14.849797 23206 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:43:14.851172 23206 webserver.cc:460] Webserver started at http://127.5.213.193:36691/ using document root <none> and password file <none>
I20260618 22:43:14.851822 23206 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:14.851926 23206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:14.852218 23206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:14.855104 23206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "9c805fd0ebcd405a814dd487626fb53d"
format_stamp: "Formatted at 2026-06-18 22:43:14 on dist-test-slave-lnj3"
server_key: "44405499973df5df58d909e6363db297"
server_key_iv: "5e545bce45a5ec6690da44b7a1197e98"
server_key_version: "encryptionkey@0"
I20260618 22:43:14.855859 23206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "9c805fd0ebcd405a814dd487626fb53d"
format_stamp: "Formatted at 2026-06-18 22:43:14 on dist-test-slave-lnj3"
server_key: "44405499973df5df58d909e6363db297"
server_key_iv: "5e545bce45a5ec6690da44b7a1197e98"
server_key_version: "encryptionkey@0"
I20260618 22:43:14.860805 23206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:43:14.864050 23221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:14.865342 23206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260618 22:43:14.865480 23206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "9c805fd0ebcd405a814dd487626fb53d"
format_stamp: "Formatted at 2026-06-18 22:43:14 on dist-test-slave-lnj3"
server_key: "44405499973df5df58d909e6363db297"
server_key_iv: "5e545bce45a5ec6690da44b7a1197e98"
server_key_version: "encryptionkey@0"
I20260618 22:43:14.865651 23206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:14.885924 23206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:14.890182 23206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:14.890456 23206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:14.891475 23206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:43:14.892907 23206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:43:14.892979 23206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:14.893095 23206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:43:14.893153 23206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:14.923152 23206 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:44593
I20260618 22:43:14.923205 23333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:44593 every 8 connection(s)
I20260618 22:43:14.924377 23206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:43:14.934396 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 23206
I20260618 22:43:14.934592 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:43:14.935248 5975 external_mini_cluster.cc:1428] Setting key 6e6a7eb3bd17dff572f323cc1c1798bd
Jun 18 22:43:14 dist-test-slave-lnj3 krb5kdc[23100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822594, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:43:14.943890 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:14.926852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:44623 (local address 127.5.213.254:46213)
0618 22:43:14.927274 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:14.927279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:14.928305 (+ 1026us) server_negotiation.cc:407] Connection header received
0618 22:43:14.930099 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:14.930105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:14.930200 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:14.930298 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:14.931834 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:14.932539 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:14.934029 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:14.934448 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:14.934561 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:14.938215 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:14.938222 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:14.938227 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:14.938264 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:14.941120 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:14.941762 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:14.941767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:14.941771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:14.941913 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:14.942276 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:14.942280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:14.942283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:14.942601 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:14.942763 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:14.943360 (+ 597us) server_negotiation.cc:299] Negotiation successful
0618 22:43:14.943568 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":209,"threads_started":1}
I20260618 22:43:14.944657 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:14.927433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:46213 (local address 127.5.213.193:44623)
0618 22:43:14.928142 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:14.928180 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:14.929828 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:14.930548 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:14.930557 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:14.931105 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:43:14.931658 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:14.931676 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:14.932673 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:14.932677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:14.933892 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:14.933905 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:14.934702 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:14.934711 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:14.934917 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:14.935561 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:43:14.935587 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:43:14.937989 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:43:14.941245 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:43:14.941251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:43:14.941264 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:43:14.941649 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:43:14.942001 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:43:14.942004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:43:14.942006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:43:14.942185 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:43:14.942727 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:43:14.942732 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:43:14.943020 (+ 288us) client_negotiation.cc:769] Sending connection context
0618 22:43:14.943245 (+ 225us) client_negotiation.cc:240] Negotiation successful
0618 22:43:14.943603 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":285,"threads_started":1}
I20260618 22:43:14.946195 23334 heartbeater.cc:347] Connected to a master server at 127.5.213.254:46213
I20260618 22:43:14.946475 23334 heartbeater.cc:464] Registering TS with master...
I20260618 22:43:14.947201 23334 heartbeater.cc:511] Master 127.5.213.254:46213 requested a full tablet report, sending...
I20260618 22:43:14.949342 23145 ts_manager.cc:194] Registered new tserver with Master: 9c805fd0ebcd405a814dd487626fb53d (127.5.213.193:44593)
I20260618 22:43:14.951174 23145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:44623
WARNING: no policy specified for kudu/127.5.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
I20260618 22:43:14.985677 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:46213
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:39215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:15.297557 23340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:15.297825 23340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:15.297957 23340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:15.305269 23340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:15.305393 23340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:15.305553 23340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:43:15.315481 23340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:39215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:46213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:15.316769 23340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:15.318049 23340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:15.330433 23346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:15.330471 23345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:15.330473 23348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:15.331038 23340 server_base.cc:1034] running on GCE node
I20260618 22:43:15.331740 23340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:15.332617 23340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:15.333835 23340 hybrid_clock.cc:648] HybridClock initialized: now 1781822595333776 us; error 50 us; skew 500 ppm
I20260618 22:43:15.334141 23340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:15 dist-test-slave-lnj3 krb5kdc[23100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822595, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:15.337325 23340 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:43:15.338792 23340 webserver.cc:460] Webserver started at http://127.5.213.194:44051/ using document root <none> and password file <none>
I20260618 22:43:15.339428 23340 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:15.339540 23340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:15.339848 23340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:15.342588 23340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "e3fa2beb4e6c4934acbc35bf5cee20ea"
format_stamp: "Formatted at 2026-06-18 22:43:15 on dist-test-slave-lnj3"
server_key: "0e15e017d4c8ff6f03474a9fef680bda"
server_key_iv: "c02c241a9f744e06166dbeb5b5c681c2"
server_key_version: "encryptionkey@0"
I20260618 22:43:15.343389 23340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "e3fa2beb4e6c4934acbc35bf5cee20ea"
format_stamp: "Formatted at 2026-06-18 22:43:15 on dist-test-slave-lnj3"
server_key: "0e15e017d4c8ff6f03474a9fef680bda"
server_key_iv: "c02c241a9f744e06166dbeb5b5c681c2"
server_key_version: "encryptionkey@0"
I20260618 22:43:15.348448 23340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:43:15.352252 23355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:15.353657 23340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:43:15.353814 23340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "e3fa2beb4e6c4934acbc35bf5cee20ea"
format_stamp: "Formatted at 2026-06-18 22:43:15 on dist-test-slave-lnj3"
server_key: "0e15e017d4c8ff6f03474a9fef680bda"
server_key_iv: "c02c241a9f744e06166dbeb5b5c681c2"
server_key_version: "encryptionkey@0"
I20260618 22:43:15.353993 23340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:15.372826 23340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:15.376842 23340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:15.377089 23340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:15.377985 23340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:43:15.379257 23340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:43:15.379333 23340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:15.379447 23340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:43:15.379503 23340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:15.410486 23340 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:39141
I20260618 22:43:15.410526 23467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:39141 every 8 connection(s)
I20260618 22:43:15.411739 23340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:43:15.417304 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 23340
I20260618 22:43:15.417452 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:43:15.417870 5975 external_mini_cluster.cc:1428] Setting key 243fca3dfee2d545296d60b5c54221f0
Jun 18 22:43:15 dist-test-slave-lnj3 krb5kdc[23100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822595, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:43:15.430104 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:15.414094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:43401 (local address 127.5.213.254:46213)
0618 22:43:15.414259 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:15.414264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:15.415566 (+ 1302us) server_negotiation.cc:407] Connection header received
0618 22:43:15.417325 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:15.417331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:15.417420 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:15.417506 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:15.419001 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:15.419743 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:15.421170 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:15.421570 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:15.421682 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:15.424790 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:15.424795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:15.424798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:15.424829 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:15.427310 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:15.428049 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:15.428053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:15.428055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:15.428189 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:15.428577 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:15.428580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:15.428582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:15.428828 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:15.428968 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:15.429709 (+ 741us) server_negotiation.cc:299] Negotiation successful
0618 22:43:15.429874 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:43:15.431349 23470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:15.414601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:46213 (local address 127.5.213.194:43401)
0618 22:43:15.415413 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:15.415464 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:15.417064 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:15.417754 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:15.417762 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:15.418269 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:43:15.418839 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:15.418858 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:15.419877 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:15.419880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:15.421042 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:15.421053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:15.421771 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:15.421777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:15.421985 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:15.422567 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:43:15.422587 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:43:15.424587 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:43:15.427428 (+ 2841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:43:15.427436 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:43:15.427452 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:43:15.427936 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:43:15.428286 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:43:15.428291 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:43:15.428295 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:43:15.428471 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:43:15.428943 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:43:15.428950 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:43:15.429321 (+ 371us) client_negotiation.cc:769] Sending connection context
0618 22:43:15.429593 (+ 272us) client_negotiation.cc:240] Negotiation successful
0618 22:43:15.430064 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":258,"threads_started":1}
I20260618 22:43:15.432696 23468 heartbeater.cc:347] Connected to a master server at 127.5.213.254:46213
I20260618 22:43:15.432988 23468 heartbeater.cc:464] Registering TS with master...
I20260618 22:43:15.433769 23468 heartbeater.cc:511] Master 127.5.213.254:46213 requested a full tablet report, sending...
I20260618 22:43:15.435340 23145 ts_manager.cc:194] Registered new tserver with Master: e3fa2beb4e6c4934acbc35bf5cee20ea (127.5.213.194:39141)
I20260618 22:43:15.436113 23145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:43401
WARNING: no policy specified for kudu/127.5.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
I20260618 22:43:15.470045 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:46213
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:39215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:15.790037 23473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:15.790364 23473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:15.790514 23473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:15.798190 23473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:15.798352 23473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:15.798529 23473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:43:15.808979 23473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:39215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:46213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:15.810276 23473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:15.811740 23473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:15.824875 23478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:15.824934 23479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:15.824955 23481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:15.825963 23473 server_base.cc:1034] running on GCE node
I20260618 22:43:15.826596 23473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:15.827692 23473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:15.828900 23473 hybrid_clock.cc:648] HybridClock initialized: now 1781822595828832 us; error 48 us; skew 500 ppm
I20260618 22:43:15.829252 23473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:15 dist-test-slave-lnj3 krb5kdc[23100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822595, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:15.832649 23473 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:43:15.834131 23473 webserver.cc:460] Webserver started at http://127.5.213.195:37423/ using document root <none> and password file <none>
I20260618 22:43:15.834858 23473 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:15.834972 23473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:15.835263 23473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:15.838249 23473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "05f08205adf34b4d86dac026a940ce78"
format_stamp: "Formatted at 2026-06-18 22:43:15 on dist-test-slave-lnj3"
server_key: "80a4f85d8a82780b37aeb10cbfceeb4e"
server_key_iv: "1990e225c136308360c72cc66da670fd"
server_key_version: "encryptionkey@0"
I20260618 22:43:15.839109 23473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "05f08205adf34b4d86dac026a940ce78"
format_stamp: "Formatted at 2026-06-18 22:43:15 on dist-test-slave-lnj3"
server_key: "80a4f85d8a82780b37aeb10cbfceeb4e"
server_key_iv: "1990e225c136308360c72cc66da670fd"
server_key_version: "encryptionkey@0"
I20260618 22:43:15.844372 23473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260618 22:43:15.847892 23488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:15.849331 23473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:43:15.849484 23473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "05f08205adf34b4d86dac026a940ce78"
format_stamp: "Formatted at 2026-06-18 22:43:15 on dist-test-slave-lnj3"
server_key: "80a4f85d8a82780b37aeb10cbfceeb4e"
server_key_iv: "1990e225c136308360c72cc66da670fd"
server_key_version: "encryptionkey@0"
I20260618 22:43:15.849712 23473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:15.876803 23473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:15.881480 23473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:15.881814 23473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:15.882916 23473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:43:15.884404 23473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:43:15.884503 23473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:15.884629 23473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:43:15.884683 23473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:15.916332 23473 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:33837
I20260618 22:43:15.916378 23600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:33837 every 8 connection(s)
I20260618 22:43:15.917657 23473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:43:15.921854 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 23473
I20260618 22:43:15.922045 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:43:15.922466 5975 external_mini_cluster.cc:1428] Setting key aa8ed277a0a852211d849b2695e4c164
Jun 18 22:43:15 dist-test-slave-lnj3 krb5kdc[23100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822595, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:43:15.938346 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:15.920136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:37045 (local address 127.5.213.254:46213)
0618 22:43:15.920336 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:15.920343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:15.921576 (+ 1233us) server_negotiation.cc:407] Connection header received
0618 22:43:15.923992 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:15.923998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:15.924101 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:15.924222 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:15.926116 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:15.927005 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:15.928886 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:15.929293 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:15.929385 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:15.932628 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:15.932633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:15.932637 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:15.932668 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:15.935039 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:15.935774 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:15.935779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:15.935783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:15.935928 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:15.936468 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:15.936474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:15.936477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:15.936871 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:15.937028 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:15.937764 (+ 736us) server_negotiation.cc:299] Negotiation successful
0618 22:43:15.937995 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:43:15.939277 23603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:15.920737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:46213 (local address 127.5.213.195:37045)
0618 22:43:15.921432 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:15.921476 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:15.923609 (+ 2133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:15.924445 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:15.924457 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:15.925158 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:43:15.925923 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:15.925952 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:15.927150 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:15.927156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:15.928754 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:15.928770 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:15.929451 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:15.929456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:15.929633 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:15.930255 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:43:15.930276 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:43:15.932425 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:43:15.935184 (+ 2759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:43:15.935191 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:43:15.935202 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:43:15.935657 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:43:15.936073 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:43:15.936079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:43:15.936082 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:43:15.936348 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:43:15.937029 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:43:15.937036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:43:15.937403 (+ 367us) client_negotiation.cc:769] Sending connection context
0618 22:43:15.937663 (+ 260us) client_negotiation.cc:240] Negotiation successful
0618 22:43:15.938157 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":276,"threads_started":1}
I20260618 22:43:15.940472 23601 heartbeater.cc:347] Connected to a master server at 127.5.213.254:46213
I20260618 22:43:15.940728 23601 heartbeater.cc:464] Registering TS with master...
I20260618 22:43:15.941365 23601 heartbeater.cc:511] Master 127.5.213.254:46213 requested a full tablet report, sending...
I20260618 22:43:15.942620 23145 ts_manager.cc:194] Registered new tserver with Master: 05f08205adf34b4d86dac026a940ce78 (127.5.213.195:33837)
I20260618 22:43:15.943414 23145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:37045
I20260618 22:43:15.950065 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:43:15.954052 23334 heartbeater.cc:503] Master 127.5.213.254:46213 was elected leader, sending a full tablet report...
I20260618 22:43:15.966976 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:15.954526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45718 (local address 127.5.213.254:46213)
0618 22:43:15.954724 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:15.954730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:15.955222 (+ 492us) server_negotiation.cc:407] Connection header received
0618 22:43:15.955415 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:15.955421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:15.955501 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:15.955581 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:15.957126 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:15.958061 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:15.959526 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:15.959950 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:15.960048 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:15.962303 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:15.962308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:15.962311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:15.962342 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:15.964644 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:15.965312 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:15.965315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:15.965318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:15.965443 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:15.965786 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:15.965789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:15.965791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:15.966067 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:15.966241 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:15.966548 (+ 307us) server_negotiation.cc:299] Negotiation successful
0618 22:43:15.966738 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:43:15.971148 23145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:43:15.977837 23145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:43:16.017349 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:15.998183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37054 (local address 127.5.213.193:44593)
0618 22:43:16.010208 (+ 12025us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:16.010217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:16.010244 (+ 27us) server_negotiation.cc:407] Connection header received
0618 22:43:16.010332 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:16.010339 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:16.010589 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:16.010793 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:16.011422 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.012793 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.015189 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.016202 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.016330 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:16.016563 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:16.016863 (+ 300us) server_negotiation.cc:299] Negotiation successful
0618 22:43:16.017022 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11856,"thread_start_us":11686,"threads_started":1}
I20260618 22:43:16.017403 23614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:15.992409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:33837 (local address 127.0.0.1:49554)
0618 22:43:15.995216 (+ 2807us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:15.995268 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:15.995458 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:16.010064 (+ 14606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:16.010072 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:16.010115 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:16.010504 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:16.010517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.012188 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.012193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:16.014088 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:16.014102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.016663 (+ 2561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.016670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:16.016832 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:16.016865 (+ 33us) client_negotiation.cc:769] Sending connection context
0618 22:43:16.016921 (+ 56us) client_negotiation.cc:240] Negotiation successful
0618 22:43:16.017046 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2513,"thread_start_us":2371,"threads_started":1}
I20260618 22:43:16.017441 23617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:15.997851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:44593 (local address 127.0.0.1:37054)
0618 22:43:16.007499 (+ 9648us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:16.007519 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:16.007653 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:16.010791 (+ 3138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:16.010794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:16.010819 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:16.011238 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:16.011250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.012916 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.012921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:16.015042 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:16.015055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.016405 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.016411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:16.016653 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:16.016695 (+ 42us) client_negotiation.cc:769] Sending connection context
0618 22:43:16.016803 (+ 108us) client_negotiation.cc:240] Negotiation successful
0618 22:43:16.016985 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9489,"mutex_wait_us":2,"thread_start_us":9468,"threads_started":1}
I20260618 22:43:16.018435 23615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:15.993005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49554 (local address 127.5.213.195:33837)
0618 22:43:16.009029 (+ 16024us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:16.009040 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:16.009082 (+ 42us) server_negotiation.cc:407] Connection header received
0618 22:43:16.009205 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:16.009215 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:16.009531 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:16.009739 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:16.010739 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.012059 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.014271 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.016419 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.017655 (+ 1236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:16.017872 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:16.017945 (+ 73us) server_negotiation.cc:299] Negotiation successful
0618 22:43:16.018094 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15653,"thread_start_us":15491,"threads_started":1}
I20260618 22:43:16.019275 23616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:15.992445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:39141 (local address 127.0.0.1:56960)
0618 22:43:16.007034 (+ 14589us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:16.007056 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:16.007213 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:16.012391 (+ 5178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:16.012395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:16.012420 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:16.012801 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:16.012812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.014658 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.014664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:16.016768 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:16.016781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.018502 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.018508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:16.018710 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:16.018742 (+ 32us) client_negotiation.cc:769] Sending connection context
0618 22:43:16.018810 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 22:43:16.018926 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14456,"mutex_wait_us":29,"thread_start_us":14114,"threads_started":1}
I20260618 22:43:16.020298 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:15.993007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56960 (local address 127.5.213.194:39141)
0618 22:43:16.011654 (+ 18647us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:16.011663 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:16.011691 (+ 28us) server_negotiation.cc:407] Connection header received
0618 22:43:16.011801 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:16.011809 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:16.012006 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:16.012384 (+ 378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:16.012972 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.014484 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.016942 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.018287 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.019606 (+ 1319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:16.019784 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:16.019850 (+ 66us) server_negotiation.cc:299] Negotiation successful
0618 22:43:16.020007 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18478,"thread_start_us":19126,"threads_started":1}
I20260618 22:43:16.021436 23269 tablet_service.cc:1467] Processing CreateTablet for tablet 2c51bef2b8b9433e940691f2e6e1cd09 (DEFAULT_TABLE table=test-table [id=adf26e4d673942f9a8540929521b9ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:43:16.022976 23269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c51bef2b8b9433e940691f2e6e1cd09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:16.023022 23403 tablet_service.cc:1467] Processing CreateTablet for tablet 2c51bef2b8b9433e940691f2e6e1cd09 (DEFAULT_TABLE table=test-table [id=adf26e4d673942f9a8540929521b9ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:43:16.023402 23536 tablet_service.cc:1467] Processing CreateTablet for tablet 2c51bef2b8b9433e940691f2e6e1cd09 (DEFAULT_TABLE table=test-table [id=adf26e4d673942f9a8540929521b9ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:43:16.024147 23403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c51bef2b8b9433e940691f2e6e1cd09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:16.024677 23536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c51bef2b8b9433e940691f2e6e1cd09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:16.034886 23619 tablet_bootstrap.cc:492] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea: Bootstrap starting.
I20260618 22:43:16.037000 23620 tablet_bootstrap.cc:492] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d: Bootstrap starting.
I20260618 22:43:16.039489 23619 tablet_bootstrap.cc:654] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:16.040307 23620 tablet_bootstrap.cc:654] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:16.040882 23619 log.cc:824] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:16.041483 23620 log.cc:824] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:16.043256 23621 tablet_bootstrap.cc:492] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78: Bootstrap starting.
I20260618 22:43:16.044091 23619 tablet_bootstrap.cc:492] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea: No bootstrap required, opened a new log
I20260618 22:43:16.044282 23620 tablet_bootstrap.cc:492] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d: No bootstrap required, opened a new log
I20260618 22:43:16.044353 23619 ts_tablet_manager.cc:1397] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260618 22:43:16.044514 23620 ts_tablet_manager.cc:1397] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260618 22:43:16.046092 23621 tablet_bootstrap.cc:654] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:16.047155 23621 log.cc:824] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:16.049280 23621 tablet_bootstrap.cc:492] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78: No bootstrap required, opened a new log
I20260618 22:43:16.049490 23621 ts_tablet_manager.cc:1397] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:43:16.054517 23620 raft_consensus.cc:348] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c805fd0ebcd405a814dd487626fb53d" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44593 } } peers { permanent_uuid: "e3fa2beb4e6c4934acbc35bf5cee20ea" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 39141 } } peers { permanent_uuid: "05f08205adf34b4d86dac026a940ce78" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33837 } }
I20260618 22:43:16.054543 23619 raft_consensus.cc:348] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c805fd0ebcd405a814dd487626fb53d" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44593 } } peers { permanent_uuid: "e3fa2beb4e6c4934acbc35bf5cee20ea" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 39141 } } peers { permanent_uuid: "05f08205adf34b4d86dac026a940ce78" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33837 } }
I20260618 22:43:16.054986 23619 raft_consensus.cc:374] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:16.054983 23620 raft_consensus.cc:374] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:16.055080 23619 raft_consensus.cc:729] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fa2beb4e6c4934acbc35bf5cee20ea, State: Initialized, Role: FOLLOWER
I20260618 22:43:16.055084 23620 raft_consensus.cc:729] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c805fd0ebcd405a814dd487626fb53d, State: Initialized, Role: FOLLOWER
I20260618 22:43:16.055694 23620 consensus_queue.cc:260] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c805fd0ebcd405a814dd487626fb53d" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44593 } } peers { permanent_uuid: "e3fa2beb4e6c4934acbc35bf5cee20ea" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 39141 } } peers { permanent_uuid: "05f08205adf34b4d86dac026a940ce78" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33837 } }
I20260618 22:43:16.055694 23619 consensus_queue.cc:260] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c805fd0ebcd405a814dd487626fb53d" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44593 } } peers { permanent_uuid: "e3fa2beb4e6c4934acbc35bf5cee20ea" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 39141 } } peers { permanent_uuid: "05f08205adf34b4d86dac026a940ce78" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33837 } }
I20260618 22:43:16.057067 23621 raft_consensus.cc:348] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c805fd0ebcd405a814dd487626fb53d" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44593 } } peers { permanent_uuid: "e3fa2beb4e6c4934acbc35bf5cee20ea" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 39141 } } peers { permanent_uuid: "05f08205adf34b4d86dac026a940ce78" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33837 } }
I20260618 22:43:16.057456 23621 raft_consensus.cc:374] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:16.057552 23621 raft_consensus.cc:729] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f08205adf34b4d86dac026a940ce78, State: Initialized, Role: FOLLOWER
I20260618 22:43:16.058175 23468 heartbeater.cc:503] Master 127.5.213.254:46213 was elected leader, sending a full tablet report...
I20260618 22:43:16.059305 23620 ts_tablet_manager.cc:1428] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260618 22:43:16.059540 23621 consensus_queue.cc:260] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c805fd0ebcd405a814dd487626fb53d" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44593 } } peers { permanent_uuid: "e3fa2beb4e6c4934acbc35bf5cee20ea" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 39141 } } peers { permanent_uuid: "05f08205adf34b4d86dac026a940ce78" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33837 } }
I20260618 22:43:16.060079 23619 ts_tablet_manager.cc:1428] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260618 22:43:16.060977 23601 heartbeater.cc:503] Master 127.5.213.254:46213 was elected leader, sending a full tablet report...
I20260618 22:43:16.061584 23621 ts_tablet_manager.cc:1428] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260618 22:43:16.065599 23625 raft_consensus.cc:482] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:43:16.065828 23625 raft_consensus.cc:504] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c805fd0ebcd405a814dd487626fb53d" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44593 } } peers { permanent_uuid: "e3fa2beb4e6c4934acbc35bf5cee20ea" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 39141 } } peers { permanent_uuid: "05f08205adf34b4d86dac026a940ce78" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33837 } }
I20260618 22:43:16.067278 23625 leader_election.cc:288] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c805fd0ebcd405a814dd487626fb53d (127.5.213.193:44593), 05f08205adf34b4d86dac026a940ce78 (127.5.213.195:33837)
I20260618 22:43:16.073973 23615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:16.068017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:45039 (local address 127.5.213.195:33837)
0618 22:43:16.068181 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:16.068186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:16.068743 (+ 557us) server_negotiation.cc:407] Connection header received
0618 22:43:16.068886 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:16.068891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:16.068955 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:16.069044 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:16.069754 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.070613 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.072323 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.073067 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.073202 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:16.073275 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:16.073607 (+ 332us) server_negotiation.cc:299] Negotiation successful
0618 22:43:16.073736 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:43:16.074146 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:16.067646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:33837 (local address 127.5.213.194:45039)
0618 22:43:16.068635 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:16.068667 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:16.068780 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:16.069094 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:16.069099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:16.069123 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:16.069566 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:16.069579 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.070762 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.070767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:16.072184 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:16.072196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.073306 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.073314 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:16.073454 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:16.073493 (+ 39us) client_negotiation.cc:769] Sending connection context
0618 22:43:16.073567 (+ 74us) client_negotiation.cc:240] Negotiation successful
0618 22:43:16.073727 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":31,"thread_start_us":696,"threads_started":1}
I20260618 22:43:16.074263 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:16.067597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:44593 (local address 127.5.213.194:54339)
0618 22:43:16.068341 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:16.068355 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:16.068499 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:16.068890 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:16.068894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:16.068919 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:16.069334 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:16.069347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.070602 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.070607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:16.072179 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:16.072192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.073322 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.073329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:16.073491 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:16.073513 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:43:16.073590 (+ 77us) client_negotiation.cc:240] Negotiation successful
0618 22:43:16.073754 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":54,"thread_start_us":417,"threads_started":1}
I20260618 22:43:16.073992 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:16.067578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:54339 (local address 127.5.213.193:44593)
0618 22:43:16.067778 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:16.067793 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:16.068440 (+ 647us) server_negotiation.cc:407] Connection header received
0618 22:43:16.068711 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:16.068716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:16.068791 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:16.068873 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:16.069456 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.070488 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.072323 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.073082 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.073231 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:16.073303 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:16.073629 (+ 326us) server_negotiation.cc:299] Negotiation successful
0618 22:43:16.073757 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:43:16.075099 23289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c51bef2b8b9433e940691f2e6e1cd09" candidate_uuid: "e3fa2beb4e6c4934acbc35bf5cee20ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c805fd0ebcd405a814dd487626fb53d" is_pre_election: true
I20260618 22:43:16.075100 23556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c51bef2b8b9433e940691f2e6e1cd09" candidate_uuid: "e3fa2beb4e6c4934acbc35bf5cee20ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f08205adf34b4d86dac026a940ce78" is_pre_election: true
I20260618 22:43:16.075531 23289 raft_consensus.cc:2454] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fa2beb4e6c4934acbc35bf5cee20ea in term 0.
I20260618 22:43:16.075536 23556 raft_consensus.cc:2454] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fa2beb4e6c4934acbc35bf5cee20ea in term 0.
I20260618 22:43:16.076082 23356 leader_election.cc:302] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c805fd0ebcd405a814dd487626fb53d, e3fa2beb4e6c4934acbc35bf5cee20ea; no voters:
I20260618 22:43:16.076414 23625 raft_consensus.cc:2781] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:43:16.076530 23625 raft_consensus.cc:482] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:43:16.076666 23625 raft_consensus.cc:3037] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:16.078758 23625 raft_consensus.cc:504] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c805fd0ebcd405a814dd487626fb53d" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44593 } } peers { permanent_uuid: "e3fa2beb4e6c4934acbc35bf5cee20ea" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 39141 } } peers { permanent_uuid: "05f08205adf34b4d86dac026a940ce78" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33837 } }
I20260618 22:43:16.079303 23625 leader_election.cc:288] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [CANDIDATE]: Term 1 election: Requested vote from peers 9c805fd0ebcd405a814dd487626fb53d (127.5.213.193:44593), 05f08205adf34b4d86dac026a940ce78 (127.5.213.195:33837)
I20260618 22:43:16.079645 23289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c51bef2b8b9433e940691f2e6e1cd09" candidate_uuid: "e3fa2beb4e6c4934acbc35bf5cee20ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c805fd0ebcd405a814dd487626fb53d"
I20260618 22:43:16.079751 23556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c51bef2b8b9433e940691f2e6e1cd09" candidate_uuid: "e3fa2beb4e6c4934acbc35bf5cee20ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f08205adf34b4d86dac026a940ce78"
I20260618 22:43:16.079834 23289 raft_consensus.cc:3037] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:16.079926 23556 raft_consensus.cc:3037] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:16.082872 23556 raft_consensus.cc:2454] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fa2beb4e6c4934acbc35bf5cee20ea in term 1.
I20260618 22:43:16.082872 23289 raft_consensus.cc:2454] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fa2beb4e6c4934acbc35bf5cee20ea in term 1.
I20260618 22:43:16.083235 23356 leader_election.cc:302] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05f08205adf34b4d86dac026a940ce78, 9c805fd0ebcd405a814dd487626fb53d, e3fa2beb4e6c4934acbc35bf5cee20ea; no voters:
I20260618 22:43:16.083525 23625 raft_consensus.cc:2781] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:16.083948 23625 raft_consensus.cc:686] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [term 1 LEADER]: Becoming Leader. State: Replica: e3fa2beb4e6c4934acbc35bf5cee20ea, State: Running, Role: LEADER
I20260618 22:43:16.084344 23625 consensus_queue.cc:237] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c805fd0ebcd405a814dd487626fb53d" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44593 } } peers { permanent_uuid: "e3fa2beb4e6c4934acbc35bf5cee20ea" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 39141 } } peers { permanent_uuid: "05f08205adf34b4d86dac026a940ce78" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33837 } }
I20260618 22:43:16.088670 23145 catalog_manager.cc:5495] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fa2beb4e6c4934acbc35bf5cee20ea (127.5.213.194). New cstate: current_term: 1 leader_uuid: "e3fa2beb4e6c4934acbc35bf5cee20ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c805fd0ebcd405a814dd487626fb53d" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3fa2beb4e6c4934acbc35bf5cee20ea" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 39141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f08205adf34b4d86dac026a940ce78" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33837 } health_report { overall_health: UNKNOWN } } }
I20260618 22:43:16.117115 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:16.111604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56974 (local address 127.5.213.194:39141)
0618 22:43:16.111822 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:16.111828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:16.111877 (+ 49us) server_negotiation.cc:407] Connection header received
0618 22:43:16.112049 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:16.112054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:16.112129 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:16.112247 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:43:16.112623 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.113342 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.114881 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.115404 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.115500 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:16.116016 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:43:16.116555 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:43:16.116678 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:16.116774 (+ 96us) server_negotiation.cc:299] Negotiation successful
0618 22:43:16.116845 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260618 22:43:16.125027 23556 raft_consensus.cc:1261] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78 [term 1 FOLLOWER]: Refusing update from remote peer e3fa2beb4e6c4934acbc35bf5cee20ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:43:16.125026 23289 raft_consensus.cc:1261] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d [term 1 FOLLOWER]: Refusing update from remote peer e3fa2beb4e6c4934acbc35bf5cee20ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:43:16.125942 23630 consensus_queue.cc:1035] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f08205adf34b4d86dac026a940ce78" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:43:16.126299 23625 consensus_queue.cc:1035] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c805fd0ebcd405a814dd487626fb53d" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 44593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:43:16.146734 23635 mvcc.cc:205] Tried to move back new op lower bound from 7298345353718517760 to 7298345353565196288. Current Snapshot: MvccSnapshot[applied={T|T < 7298345353718517760 or (T in {7298345353718517760})}]
I20260618 22:43:16.153460 23144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:43:16.153739 23144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:43:16.157452 23144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c135fe2858b641d799364cd20afe95a2: Sending DeleteTablet for 3 replicas of tablet 2c51bef2b8b9433e940691f2e6e1cd09
I20260618 22:43:16.158605 23536 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c51bef2b8b9433e940691f2e6e1cd09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:43:16 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:49554
I20260618 22:43:16.158721 23403 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c51bef2b8b9433e940691f2e6e1cd09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:43:16 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:56960
I20260618 22:43:16.159490 23643 tablet_replica.cc:330] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78: stopping tablet replica
I20260618 22:43:16.159409 23269 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c51bef2b8b9433e940691f2e6e1cd09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:43:16 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:37054
I20260618 22:43:16.160008 23643 raft_consensus.cc:2229] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:43:16.160548 23643 raft_consensus.cc:2258] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:43:16.161337 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 23206
I20260618 22:43:16.165498 23645 tablet_replica.cc:330] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d: stopping tablet replica
I20260618 22:43:16.165943 23645 raft_consensus.cc:2229] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:43:16.166442 23645 raft_consensus.cc:2258] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:43:16.169229 23645 ts_tablet_manager.cc:1905] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:43:16.170447 23643 ts_tablet_manager.cc:1905] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 18 22:43:16 dist-test-slave-lnj3 krb5kdc[23100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:43:16.177333 23645 ts_tablet_manager.cc:1918] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:43:16.177493 23645 log.cc:1196] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/2c51bef2b8b9433e940691f2e6e1cd09
I20260618 22:43:16.178012 23645 ts_tablet_manager.cc:1939] T 2c51bef2b8b9433e940691f2e6e1cd09 P 9c805fd0ebcd405a814dd487626fb53d: Deleting consensus metadata
I20260618 22:43:16.180003 23129 catalog_manager.cc:4841] TS 9c805fd0ebcd405a814dd487626fb53d (127.5.213.193:44593): tablet 2c51bef2b8b9433e940691f2e6e1cd09 (table test-table [id=adf26e4d673942f9a8540929521b9ff5]) successfully deleted
I20260618 22:43:16.182112 23643 ts_tablet_manager.cc:1918] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:43:16.182282 23643 log.cc:1196] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/2c51bef2b8b9433e940691f2e6e1cd09
I20260618 22:43:16.183584 23644 tablet_replica.cc:330] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea: stopping tablet replica
I20260618 22:43:16.184048 23644 raft_consensus.cc:2229] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:43:16.184288 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:16.162573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37068 (local address 127.5.213.193:44593)
0618 22:43:16.162796 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:16.162803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:16.167198 (+ 4395us) server_negotiation.cc:407] Connection header received
0618 22:43:16.167366 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:16.167371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:16.167447 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:16.167535 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:16.169216 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.170217 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.172122 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.172743 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.172831 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:16.178350 (+ 5519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:16.178363 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:16.178381 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:16.178419 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:16.181606 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:16.182370 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:16.182380 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:16.182383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:16.182518 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:16.182925 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:16.182931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:16.182934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:16.183282 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:16.183551 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:16.183847 (+ 296us) server_negotiation.cc:299] Negotiation successful
0618 22:43:16.184014 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260618 22:43:16.184732 23644 raft_consensus.cc:2258] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:43:16.185439 23329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:16.186872 23644 ts_tablet_manager.cc:1905] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:43:16.193945 23644 ts_tablet_manager.cc:1918] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:43:16.194106 23644 log.cc:1196] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/2c51bef2b8b9433e940691f2e6e1cd09
I20260618 22:43:16.194474 23643 ts_tablet_manager.cc:1939] T 2c51bef2b8b9433e940691f2e6e1cd09 P 05f08205adf34b4d86dac026a940ce78: Deleting consensus metadata
I20260618 22:43:16.194653 23644 ts_tablet_manager.cc:1939] T 2c51bef2b8b9433e940691f2e6e1cd09 P e3fa2beb4e6c4934acbc35bf5cee20ea: Deleting consensus metadata
I20260618 22:43:16.196483 23130 catalog_manager.cc:4841] TS e3fa2beb4e6c4934acbc35bf5cee20ea (127.5.213.194:39141): tablet 2c51bef2b8b9433e940691f2e6e1cd09 (table test-table [id=adf26e4d673942f9a8540929521b9ff5]) successfully deleted
I20260618 22:43:16.196563 23131 catalog_manager.cc:4841] TS 05f08205adf34b4d86dac026a940ce78 (127.5.213.195:33837): tablet 2c51bef2b8b9433e940691f2e6e1cd09 (table test-table [id=adf26e4d673942f9a8540929521b9ff5]) successfully deleted
I20260618 22:43:16.291411 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 23206
I20260618 22:43:16.306367 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 23340
Jun 18 22:43:16 dist-test-slave-lnj3 krb5kdc[23100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.194@KRBTEST.COM
I20260618 22:43:16.323551 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:16.307224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56976 (local address 127.5.213.194:39141)
0618 22:43:16.307424 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:16.307435 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:16.307456 (+ 21us) server_negotiation.cc:407] Connection header received
0618 22:43:16.307596 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:16.307602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:16.307686 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:16.307778 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:16.308968 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.310154 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.311706 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.312360 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.312486 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:16.317348 (+ 4862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:16.317370 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:16.317387 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:16.317435 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:16.320646 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:16.321447 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:16.321456 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:16.321458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:16.321628 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:16.322022 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:16.322026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:16.322028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:16.322412 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:16.322738 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:16.323011 (+ 273us) server_negotiation.cc:299] Negotiation successful
0618 22:43:16.323193 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260618 22:43:16.324451 23463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:16.437018 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 23340
I20260618 22:43:16.451959 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 23473
Jun 18 22:43:16 dist-test-slave-lnj3 krb5kdc[23100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.195@KRBTEST.COM
I20260618 22:43:16.468448 23615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:16.452630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.5.213.195:33837)
0618 22:43:16.452852 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:16.452857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:16.452874 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:43:16.452993 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:16.452998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:16.453083 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:16.453166 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:16.454447 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.455234 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.456561 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:16.456949 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:16.457035 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:16.462189 (+ 5154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:16.462208 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:16.462228 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:16.462270 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:16.465700 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:16.466399 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:16.466406 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:16.466408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:16.466577 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:16.467028 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:16.467033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:16.467037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:16.467505 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:16.467808 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:16.467976 (+ 168us) server_negotiation.cc:299] Negotiation successful
0618 22:43:16.468143 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:43:16.469120 23596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:16.557314 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 23473
I20260618 22:43:16.570302 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 23113
I20260618 22:43:16.571507 23175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:16.679421 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 23113
2026-06-18T22:43:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (3990 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-18T22:43:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:16Z Disabled control of system clock
I20260618 22:43:16.747273 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:44029
--webserver_interface=127.5.213.254
--webserver_port=0
--builtin_ntp_servers=127.5.213.212:42263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:44029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 22:43:17.061415 23654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:17.061691 23654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:17.061792 23654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:17.068921 23654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:43:17.069029 23654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:17.069072 23654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:17.069149 23654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:43:17.069209 23654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:43:17.078510 23654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:42263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:44029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:44029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:17.079897 23654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:17.081665 23654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:17.092557 23659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:17.092480 23660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:17.092530 23662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:17.093757 23654 server_base.cc:1034] running on GCE node
I20260618 22:43:17.094424 23654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:17.095762 23654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:17.096976 23654 hybrid_clock.cc:648] HybridClock initialized: now 1781822597096929 us; error 37 us; skew 500 ppm
I20260618 22:43:17.097308 23654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:43:17.099612 23654 webserver.cc:460] Webserver started at http://127.5.213.254:42493/ using document root <none> and password file <none>
I20260618 22:43:17.100258 23654 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:17.100344 23654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:17.100646 23654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:17.103474 23654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "c69abaf9d2f445c1b8d36f18d5a0f4a7"
format_stamp: "Formatted at 2026-06-18 22:43:17 on dist-test-slave-lnj3"
server_key: "71e7d80a11d937af517acedf6e859239"
server_key_iv: "fe08f7aed4904ee28d88a714541f81b4"
server_key_version: "encryptionkey@0"
I20260618 22:43:17.104244 23654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "c69abaf9d2f445c1b8d36f18d5a0f4a7"
format_stamp: "Formatted at 2026-06-18 22:43:17 on dist-test-slave-lnj3"
server_key: "71e7d80a11d937af517acedf6e859239"
server_key_iv: "fe08f7aed4904ee28d88a714541f81b4"
server_key_version: "encryptionkey@0"
I20260618 22:43:17.109112 23654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260618 22:43:17.112500 23668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:17.113839 23654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:43:17.113992 23654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "c69abaf9d2f445c1b8d36f18d5a0f4a7"
format_stamp: "Formatted at 2026-06-18 22:43:17 on dist-test-slave-lnj3"
server_key: "71e7d80a11d937af517acedf6e859239"
server_key_iv: "fe08f7aed4904ee28d88a714541f81b4"
server_key_version: "encryptionkey@0"
I20260618 22:43:17.114185 23654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:17.129015 23654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:17.129755 23654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:17.130010 23654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:17.146327 23654 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:44029
I20260618 22:43:17.146334 23719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:44029 every 8 connection(s)
I20260618 22:43:17.147656 23654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:43:17.148072 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 23654
I20260618 22:43:17.148252 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:43:17.148821 5975 external_mini_cluster.cc:1428] Setting key 5bcdf2203bf31d857b50e4f544afb813
I20260618 22:43:17.151971 23720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:17.161715 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:17.150811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56120 (local address 127.5.213.254:44029)
0618 22:43:17.151809 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:17.151819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:17.151857 (+ 38us) server_negotiation.cc:407] Connection header received
0618 22:43:17.153435 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:17.153457 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:17.153878 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:17.154248 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:17.154820 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:17.156116 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:17.157520 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:17.158159 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:17.158325 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:17.159541 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:17.159549 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:17.159563 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:17.160122 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:17.160224 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:17.160247 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:17.160353 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:17.160567 (+ 214us) server_negotiation.cc:299] Negotiation successful
0618 22:43:17.160848 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":363,"threads_started":1}
I20260618 22:43:17.162477 23720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7: Bootstrap starting.
I20260618 22:43:17.165604 23720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:17.166481 23720 log.cc:824] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:17.168999 23720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7: No bootstrap required, opened a new log
I20260618 22:43:17.175988 23720 raft_consensus.cc:348] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69abaf9d2f445c1b8d36f18d5a0f4a7" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44029 } }
I20260618 22:43:17.176294 23720 raft_consensus.cc:374] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:17.176379 23720 raft_consensus.cc:729] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c69abaf9d2f445c1b8d36f18d5a0f4a7, State: Initialized, Role: FOLLOWER
I20260618 22:43:17.176865 23720 consensus_queue.cc:260] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69abaf9d2f445c1b8d36f18d5a0f4a7" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44029 } }
I20260618 22:43:17.177064 23720 raft_consensus.cc:388] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:43:17.177158 23720 raft_consensus.cc:482] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:43:17.177277 23720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:17.179312 23720 raft_consensus.cc:504] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69abaf9d2f445c1b8d36f18d5a0f4a7" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44029 } }
I20260618 22:43:17.179808 23720 leader_election.cc:302] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c69abaf9d2f445c1b8d36f18d5a0f4a7; no voters:
I20260618 22:43:17.180366 23720 leader_election.cc:288] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:43:17.180490 23725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:17.181182 23725 raft_consensus.cc:686] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7 [term 1 LEADER]: Becoming Leader. State: Replica: c69abaf9d2f445c1b8d36f18d5a0f4a7, State: Running, Role: LEADER
I20260618 22:43:17.181646 23725 consensus_queue.cc:237] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69abaf9d2f445c1b8d36f18d5a0f4a7" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44029 } }
I20260618 22:43:17.182224 23720 sys_catalog.cc:564] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:43:17.184146 23726 sys_catalog.cc:455] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c69abaf9d2f445c1b8d36f18d5a0f4a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69abaf9d2f445c1b8d36f18d5a0f4a7" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44029 } } }
I20260618 22:43:17.185050 23726 sys_catalog.cc:458] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:17.184754 23727 sys_catalog.cc:455] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c69abaf9d2f445c1b8d36f18d5a0f4a7. Latest consensus state: current_term: 1 leader_uuid: "c69abaf9d2f445c1b8d36f18d5a0f4a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69abaf9d2f445c1b8d36f18d5a0f4a7" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 44029 } } }
I20260618 22:43:17.185521 23727 sys_catalog.cc:458] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:17.186435 23733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:43:17.191350 23733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:43:17.200639 23733 catalog_manager.cc:1347] Generated new cluster ID: 06ef3442d20f4962b03ffae1a045426b
I20260618 22:43:17.200757 23733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:43:17.233927 23733 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:43:17.235814 23733 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:43:17.255368 23733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7: Generated new TSK 0
I20260618 22:43:17.256395 23733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:43:17.263132 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44029
--builtin_ntp_servers=127.5.213.212:42263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 22:43:17.590963 23744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:17.591257 23744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:17.591355 23744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:17.598548 23744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:17.598682 23744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:17.598790 23744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:43:17.607934 23744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:42263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:17.609125 23744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:17.610878 23744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:17.623433 23749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:17.623580 23750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:17.623472 23752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:17.624716 23744 server_base.cc:1034] running on GCE node
I20260618 22:43:17.625226 23744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:17.626070 23744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:17.627251 23744 hybrid_clock.cc:648] HybridClock initialized: now 1781822597627197 us; error 41 us; skew 500 ppm
I20260618 22:43:17.627518 23744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:43:17.634894 23744 webserver.cc:460] Webserver started at http://127.5.213.193:40611/ using document root <none> and password file <none>
I20260618 22:43:17.635567 23744 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:17.635664 23744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:17.635948 23744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:17.638846 23744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "794a373e8fa545c0bb05aa47f0abb056"
format_stamp: "Formatted at 2026-06-18 22:43:17 on dist-test-slave-lnj3"
server_key: "cd1e563b6dfbb9cc6b6778e407da07ab"
server_key_iv: "634f3aeabebe4742b2fdd743451d171c"
server_key_version: "encryptionkey@0"
I20260618 22:43:17.639585 23744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "794a373e8fa545c0bb05aa47f0abb056"
format_stamp: "Formatted at 2026-06-18 22:43:17 on dist-test-slave-lnj3"
server_key: "cd1e563b6dfbb9cc6b6778e407da07ab"
server_key_iv: "634f3aeabebe4742b2fdd743451d171c"
server_key_version: "encryptionkey@0"
I20260618 22:43:17.644855 23744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260618 22:43:17.648447 23758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:17.649845 23744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:43:17.650002 23744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "794a373e8fa545c0bb05aa47f0abb056"
format_stamp: "Formatted at 2026-06-18 22:43:17 on dist-test-slave-lnj3"
server_key: "cd1e563b6dfbb9cc6b6778e407da07ab"
server_key_iv: "634f3aeabebe4742b2fdd743451d171c"
server_key_version: "encryptionkey@0"
I20260618 22:43:17.650223 23744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:17.671792 23744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:17.672672 23744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:17.673002 23744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:17.674075 23744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:43:17.675738 23744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:43:17.675833 23744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:17.675947 23744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:43:17.676023 23744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:17.706357 23744 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:35739
I20260618 22:43:17.706414 23870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:35739 every 8 connection(s)
I20260618 22:43:17.707724 23744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:43:17.714973 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 23744
I20260618 22:43:17.715144 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:43:17.715714 5975 external_mini_cluster.cc:1428] Setting key e7347c1147d193e6414d52ce2df02d81
I20260618 22:43:17.718744 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44029
--builtin_ntp_servers=127.5.213.212:42263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 22:43:17.721503 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:17.710505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:38053 (local address 127.5.213.254:44029)
0618 22:43:17.710969 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:17.710974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:17.712258 (+ 1284us) server_negotiation.cc:407] Connection header received
0618 22:43:17.714106 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:17.714110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:17.714189 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:17.714295 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:17.715682 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:17.716572 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:17.718583 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:17.719169 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:17.719274 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:17.720808 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:17.720813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:17.720817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:17.720835 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:17.720876 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:17.720888 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:17.720961 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:17.721176 (+ 215us) server_negotiation.cc:299] Negotiation successful
0618 22:43:17.721255 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":246,"threads_started":1}
I20260618 22:43:17.722922 23874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:17.711183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:44029 (local address 127.5.213.193:38053)
0618 22:43:17.712078 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:17.712131 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:17.713836 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:17.714473 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:17.714480 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:17.714582 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:43:17.715495 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:17.715520 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:17.716726 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:17.716732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:17.718390 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:17.718405 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:17.719390 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:17.719397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:17.719693 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:17.720445 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:43:17.720448 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:43:17.720502 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:43:17.720606 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:43:17.720975 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:43:17.720984 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:43:17.720992 (+ 8us) client_negotiation.cc:769] Sending connection context
0618 22:43:17.721169 (+ 177us) client_negotiation.cc:240] Negotiation successful
0618 22:43:17.721559 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":266,"threads_started":1}
I20260618 22:43:17.724402 23871 heartbeater.cc:347] Connected to a master server at 127.5.213.254:44029
I20260618 22:43:17.724713 23871 heartbeater.cc:464] Registering TS with master...
I20260618 22:43:17.725453 23871 heartbeater.cc:511] Master 127.5.213.254:44029 requested a full tablet report, sending...
I20260618 22:43:17.727443 23685 ts_manager.cc:194] Registered new tserver with Master: 794a373e8fa545c0bb05aa47f0abb056 (127.5.213.193:35739)
I20260618 22:43:17.728909 23685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.213.193:38053
W20260618 22:43:18.032011 23875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:18.032274 23875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:18.032382 23875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:18.039765 23875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:18.039860 23875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:18.039990 23875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:43:18.049062 23875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:42263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:18.050349 23875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:18.052083 23875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:18.064172 23883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:18.064251 23881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:18.064386 23880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:18.064819 23875 server_base.cc:1034] running on GCE node
I20260618 22:43:18.065429 23875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:18.066289 23875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:18.067489 23875 hybrid_clock.cc:648] HybridClock initialized: now 1781822598067435 us; error 44 us; skew 500 ppm
I20260618 22:43:18.067796 23875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:43:18.069957 23875 webserver.cc:460] Webserver started at http://127.5.213.194:40375/ using document root <none> and password file <none>
I20260618 22:43:18.070616 23875 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:18.070749 23875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:18.071041 23875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:18.073899 23875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "d9aa07cd2f8544a28ccd6a563c89caba"
format_stamp: "Formatted at 2026-06-18 22:43:18 on dist-test-slave-lnj3"
server_key: "02e037381945c6afb3092ddc5fd64e3c"
server_key_iv: "23825cb1c721709869fca86f190d4be1"
server_key_version: "encryptionkey@0"
I20260618 22:43:18.074662 23875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "d9aa07cd2f8544a28ccd6a563c89caba"
format_stamp: "Formatted at 2026-06-18 22:43:18 on dist-test-slave-lnj3"
server_key: "02e037381945c6afb3092ddc5fd64e3c"
server_key_iv: "23825cb1c721709869fca86f190d4be1"
server_key_version: "encryptionkey@0"
I20260618 22:43:18.079636 23875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260618 22:43:18.083105 23889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:18.084467 23875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260618 22:43:18.084616 23875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "d9aa07cd2f8544a28ccd6a563c89caba"
format_stamp: "Formatted at 2026-06-18 22:43:18 on dist-test-slave-lnj3"
server_key: "02e037381945c6afb3092ddc5fd64e3c"
server_key_iv: "23825cb1c721709869fca86f190d4be1"
server_key_version: "encryptionkey@0"
I20260618 22:43:18.084816 23875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:18.099920 23875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:18.100672 23875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:18.100930 23875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:18.101835 23875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:43:18.103250 23875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:43:18.103343 23875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:18.103449 23875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:43:18.103516 23875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:18.132310 23875 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:35085
I20260618 22:43:18.132359 24001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:35085 every 8 connection(s)
I20260618 22:43:18.133562 23875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:43:18.141602 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 23875
I20260618 22:43:18.141793 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:43:18.142217 5975 external_mini_cluster.cc:1428] Setting key 28ca1d12336fec85992307f675fc6416
I20260618 22:43:18.144841 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44029
--builtin_ntp_servers=127.5.213.212:42263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 22:43:18.146011 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.136219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:34771 (local address 127.5.213.254:44029)
0618 22:43:18.136395 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:18.136400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:18.137730 (+ 1330us) server_negotiation.cc:407] Connection header received
0618 22:43:18.139582 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:18.139588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:18.139667 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:18.139753 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:18.140827 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.141530 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.143161 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.143779 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.143906 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:18.145220 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:18.145225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:18.145229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:18.145245 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:18.145275 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:18.145286 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:18.145358 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:18.145585 (+ 227us) server_negotiation.cc:299] Negotiation successful
0618 22:43:18.145682 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":52}
I20260618 22:43:18.147166 24004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.136715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:44029 (local address 127.5.213.194:34771)
0618 22:43:18.137579 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:18.137614 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:18.139315 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:18.139992 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:18.140001 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:18.140091 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:43:18.140651 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:18.140669 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.141670 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.141674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:18.143019 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:18.143032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.144009 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.144016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:18.144258 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:18.144892 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:43:18.144895 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:43:18.144935 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:43:18.145030 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:43:18.145370 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:43:18.145381 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:43:18.145389 (+ 8us) client_negotiation.cc:769] Sending connection context
0618 22:43:18.145544 (+ 155us) client_negotiation.cc:240] Negotiation successful
0618 22:43:18.145956 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":256,"threads_started":1}
I20260618 22:43:18.148512 24002 heartbeater.cc:347] Connected to a master server at 127.5.213.254:44029
I20260618 22:43:18.148797 24002 heartbeater.cc:464] Registering TS with master...
I20260618 22:43:18.149545 24002 heartbeater.cc:511] Master 127.5.213.254:44029 requested a full tablet report, sending...
I20260618 22:43:18.150936 23685 ts_manager.cc:194] Registered new tserver with Master: d9aa07cd2f8544a28ccd6a563c89caba (127.5.213.194:35085)
I20260618 22:43:18.151767 23685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.213.194:34771
W20260618 22:43:18.466323 24005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:18.466579 24005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:18.466697 24005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:18.474193 24005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:18.474295 24005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:18.474402 24005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:43:18.483708 24005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:42263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:44029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:18.484908 24005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:18.486617 24005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:18.499229 24010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:18.499229 24013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:18.499164 24011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:18.500182 24005 server_base.cc:1034] running on GCE node
I20260618 22:43:18.500820 24005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:18.501806 24005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:18.503002 24005 hybrid_clock.cc:648] HybridClock initialized: now 1781822598502941 us; error 57 us; skew 500 ppm
I20260618 22:43:18.503315 24005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:43:18.505522 24005 webserver.cc:460] Webserver started at http://127.5.213.195:37077/ using document root <none> and password file <none>
I20260618 22:43:18.506197 24005 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:18.506299 24005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:18.506596 24005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:18.509449 24005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "4bcff14675d247fcb7098b4cc7f1e04d"
format_stamp: "Formatted at 2026-06-18 22:43:18 on dist-test-slave-lnj3"
server_key: "a8d6290bdde892329f2dd77f6887b698"
server_key_iv: "1073d9f90257c8979714292be1683004"
server_key_version: "encryptionkey@0"
I20260618 22:43:18.510208 24005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "4bcff14675d247fcb7098b4cc7f1e04d"
format_stamp: "Formatted at 2026-06-18 22:43:18 on dist-test-slave-lnj3"
server_key: "a8d6290bdde892329f2dd77f6887b698"
server_key_iv: "1073d9f90257c8979714292be1683004"
server_key_version: "encryptionkey@0"
I20260618 22:43:18.515169 24005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260618 22:43:18.518729 24019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:18.520018 24005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:43:18.520174 24005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "4bcff14675d247fcb7098b4cc7f1e04d"
format_stamp: "Formatted at 2026-06-18 22:43:18 on dist-test-slave-lnj3"
server_key: "a8d6290bdde892329f2dd77f6887b698"
server_key_iv: "1073d9f90257c8979714292be1683004"
server_key_version: "encryptionkey@0"
I20260618 22:43:18.520359 24005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:18.534714 24005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:18.535480 24005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:18.535748 24005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:18.536700 24005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:43:18.538069 24005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:43:18.538166 24005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:18.538276 24005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:43:18.538343 24005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:18.568127 24005 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:43299
I20260618 22:43:18.568169 24131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:43299 every 8 connection(s)
I20260618 22:43:18.569368 24005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:43:18.573751 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 24005
I20260618 22:43:18.573942 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:43:18.574524 5975 external_mini_cluster.cc:1428] Setting key 82fc0321f7c2b818b507fd5542ad9cb2
I20260618 22:43:18.582530 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.571927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:40703 (local address 127.5.213.254:44029)
0618 22:43:18.572133 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:18.572138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:18.573532 (+ 1394us) server_negotiation.cc:407] Connection header received
0618 22:43:18.575994 (+ 2462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:18.575998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:18.576067 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:18.576147 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:18.577399 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.578201 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.580019 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.580426 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.580516 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:18.581822 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:18.581827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:18.581830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:18.581847 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:18.581880 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:18.581891 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:18.581962 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:18.582209 (+ 247us) server_negotiation.cc:299] Negotiation successful
0618 22:43:18.582284 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260618 22:43:18.583724 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.572405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:44029 (local address 127.5.213.195:40703)
0618 22:43:18.573329 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:18.573385 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:18.575715 (+ 2330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:18.576436 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:18.576449 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:18.576587 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:43:18.577228 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:18.577250 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.578348 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.578352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:18.579885 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:18.579900 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.580655 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.580661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:18.580886 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:18.581515 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:43:18.581517 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:43:18.581556 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:43:18.581638 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:43:18.582006 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:43:18.582015 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:43:18.582022 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 22:43:18.582190 (+ 168us) client_negotiation.cc:240] Negotiation successful
0618 22:43:18.582613 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":266,"threads_started":1}
I20260618 22:43:18.584964 24132 heartbeater.cc:347] Connected to a master server at 127.5.213.254:44029
I20260618 22:43:18.585232 24132 heartbeater.cc:464] Registering TS with master...
I20260618 22:43:18.586000 24132 heartbeater.cc:511] Master 127.5.213.254:44029 requested a full tablet report, sending...
I20260618 22:43:18.587468 23685 ts_manager.cc:194] Registered new tserver with Master: 4bcff14675d247fcb7098b4cc7f1e04d (127.5.213.195:43299)
I20260618 22:43:18.588411 23685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.213.195:40703
I20260618 22:43:18.590418 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:43:18.600538 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.594766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56126 (local address 127.5.213.254:44029)
0618 22:43:18.594941 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:18.594945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:18.595353 (+ 408us) server_negotiation.cc:407] Connection header received
0618 22:43:18.595477 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:18.595480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:18.595559 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:18.595666 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:18.596106 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.596818 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.598290 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.598773 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.598866 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:18.599979 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:18.599985 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:18.599988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:18.600005 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:18.600039 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:18.600051 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:18.600133 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:18.600239 (+ 106us) server_negotiation.cc:299] Negotiation successful
0618 22:43:18.600319 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 22:43:18.604465 23685 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:43:18.611294 23685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:43:18.645277 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.623133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:43299 (local address 127.0.0.1:58052)
0618 22:43:18.636451 (+ 13318us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:18.636466 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:18.636681 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:18.637479 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:18.637486 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:18.637516 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:18.637967 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:18.637986 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.639865 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.639870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:18.642425 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:18.642440 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.644317 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.644324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:18.644501 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:18.644551 (+ 50us) client_negotiation.cc:769] Sending connection context
0618 22:43:18.644659 (+ 108us) client_negotiation.cc:240] Negotiation successful
0618 22:43:18.644860 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13167,"mutex_wait_us":99,"spinlock_wait_cycles":3328,"thread_start_us":11238,"threads_started":1}
I20260618 22:43:18.646026 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.623149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41892 (local address 127.5.213.194:35085)
0618 22:43:18.635602 (+ 12453us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:18.635611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:18.636586 (+ 975us) server_negotiation.cc:407] Connection header received
0618 22:43:18.636906 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:18.636915 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:18.637203 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:18.637387 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:18.638752 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.640144 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.642769 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.644108 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.644240 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:18.644522 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:18.645510 (+ 988us) server_negotiation.cc:299] Negotiation successful
0618 22:43:18.645700 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12261,"thread_start_us":12159,"threads_started":1}
I20260618 22:43:18.646099 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.623774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35739 (local address 127.0.0.1:53058)
0618 22:43:18.637182 (+ 13408us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:18.637203 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:18.637419 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:18.638197 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:18.638202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:18.638238 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:18.638604 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:18.638614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.639910 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.639914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:18.643637 (+ 3723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:18.643648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.645364 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.645374 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:18.645533 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:18.645563 (+ 30us) client_negotiation.cc:769] Sending connection context
0618 22:43:18.645627 (+ 64us) client_negotiation.cc:240] Negotiation successful
0618 22:43:18.645769 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13306,"mutex_wait_us":96,"thread_start_us":12375,"threads_started":1}
I20260618 22:43:18.645380 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.622881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:35085 (local address 127.0.0.1:41892)
0618 22:43:18.636405 (+ 13524us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:18.636452 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:18.636681 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:18.637660 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:18.637665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:18.637693 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:18.638505 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:18.638516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.640538 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.640543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:18.642557 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:18.642569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.644652 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.644658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:18.644856 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:18.644886 (+ 30us) client_negotiation.cc:769] Sending connection context
0618 22:43:18.644951 (+ 65us) client_negotiation.cc:240] Negotiation successful
0618 22:43:18.645065 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11692,"spinlock_wait_cycles":34688,"thread_start_us":14071,"threads_started":1}
I20260618 22:43:18.646965 24148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.623791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58052 (local address 127.5.213.195:43299)
0618 22:43:18.635168 (+ 11377us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:18.635179 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:18.636589 (+ 1410us) server_negotiation.cc:407] Connection header received
0618 22:43:18.636914 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:18.636925 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:18.637224 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:18.637776 (+ 552us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:18.638138 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.639716 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.642658 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.644117 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.645879 (+ 1762us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:18.646329 (+ 450us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:18.646402 (+ 73us) server_negotiation.cc:299] Negotiation successful
0618 22:43:18.646538 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11166,"thread_start_us":11013,"threads_started":1}
I20260618 22:43:18.647639 24149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.624328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53058 (local address 127.5.213.193:35739)
0618 22:43:18.635939 (+ 11611us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:18.635949 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:18.637321 (+ 1372us) server_negotiation.cc:407] Connection header received
0618 22:43:18.637595 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:18.637602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:18.637798 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:18.637950 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:18.638800 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.639764 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.643817 (+ 4053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.645136 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.645307 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:18.645726 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:18.645802 (+ 76us) server_negotiation.cc:299] Negotiation successful
0618 22:43:18.645957 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11454,"thread_start_us":11341,"threads_started":1}
I20260618 22:43:18.650285 24067 tablet_service.cc:1467] Processing CreateTablet for tablet d23403f3d3c4443087bc3d3a91e7593d (DEFAULT_TABLE table=test-table [id=b585e41fc18440c588fa6ee9737ad074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:43:18.650285 23937 tablet_service.cc:1467] Processing CreateTablet for tablet d23403f3d3c4443087bc3d3a91e7593d (DEFAULT_TABLE table=test-table [id=b585e41fc18440c588fa6ee9737ad074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:43:18.651859 24067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d23403f3d3c4443087bc3d3a91e7593d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:18.652249 23937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d23403f3d3c4443087bc3d3a91e7593d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:18.654132 23806 tablet_service.cc:1467] Processing CreateTablet for tablet d23403f3d3c4443087bc3d3a91e7593d (DEFAULT_TABLE table=test-table [id=b585e41fc18440c588fa6ee9737ad074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:43:18.655404 23806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d23403f3d3c4443087bc3d3a91e7593d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:18.663342 24150 tablet_bootstrap.cc:492] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba: Bootstrap starting.
I20260618 22:43:18.665871 24151 tablet_bootstrap.cc:492] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d: Bootstrap starting.
I20260618 22:43:18.666667 24150 tablet_bootstrap.cc:654] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:18.667984 24150 log.cc:824] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:18.668691 24152 tablet_bootstrap.cc:492] T d23403f3d3c4443087bc3d3a91e7593d P 794a373e8fa545c0bb05aa47f0abb056: Bootstrap starting.
I20260618 22:43:18.669329 24151 tablet_bootstrap.cc:654] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:18.670572 24151 log.cc:824] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:18.671718 24152 tablet_bootstrap.cc:654] T d23403f3d3c4443087bc3d3a91e7593d P 794a373e8fa545c0bb05aa47f0abb056: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:18.672607 24152 log.cc:824] T d23403f3d3c4443087bc3d3a91e7593d P 794a373e8fa545c0bb05aa47f0abb056: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:18.673755 24151 tablet_bootstrap.cc:492] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d: No bootstrap required, opened a new log
I20260618 22:43:18.674044 24151 ts_tablet_manager.cc:1397] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260618 22:43:18.675223 24150 tablet_bootstrap.cc:492] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba: No bootstrap required, opened a new log
I20260618 22:43:18.675449 24150 ts_tablet_manager.cc:1397] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260618 22:43:18.675472 24152 tablet_bootstrap.cc:492] T d23403f3d3c4443087bc3d3a91e7593d P 794a373e8fa545c0bb05aa47f0abb056: No bootstrap required, opened a new log
I20260618 22:43:18.675722 24152 ts_tablet_manager.cc:1397] T d23403f3d3c4443087bc3d3a91e7593d P 794a373e8fa545c0bb05aa47f0abb056: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260618 22:43:18.682685 24150 raft_consensus.cc:348] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9aa07cd2f8544a28ccd6a563c89caba" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35085 } } peers { permanent_uuid: "4bcff14675d247fcb7098b4cc7f1e04d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43299 } } peers { permanent_uuid: "794a373e8fa545c0bb05aa47f0abb056" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35739 } }
I20260618 22:43:18.683112 24150 raft_consensus.cc:374] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:18.683223 24150 raft_consensus.cc:729] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9aa07cd2f8544a28ccd6a563c89caba, State: Initialized, Role: FOLLOWER
I20260618 22:43:18.683822 24150 consensus_queue.cc:260] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9aa07cd2f8544a28ccd6a563c89caba" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35085 } } peers { permanent_uuid: "4bcff14675d247fcb7098b4cc7f1e04d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43299 } } peers { permanent_uuid: "794a373e8fa545c0bb05aa47f0abb056" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35739 } }
I20260618 22:43:18.684237 24151 raft_consensus.cc:348] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9aa07cd2f8544a28ccd6a563c89caba" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35085 } } peers { permanent_uuid: "4bcff14675d247fcb7098b4cc7f1e04d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43299 } } peers { permanent_uuid: "794a373e8fa545c0bb05aa47f0abb056" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35739 } }
I20260618 22:43:18.684674 24151 raft_consensus.cc:374] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:18.684769 24151 raft_consensus.cc:729] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcff14675d247fcb7098b4cc7f1e04d, State: Initialized, Role: FOLLOWER
I20260618 22:43:18.684971 24002 heartbeater.cc:503] Master 127.5.213.254:44029 was elected leader, sending a full tablet report...
I20260618 22:43:18.685586 24150 ts_tablet_manager.cc:1428] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260618 22:43:18.685467 24151 consensus_queue.cc:260] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9aa07cd2f8544a28ccd6a563c89caba" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35085 } } peers { permanent_uuid: "4bcff14675d247fcb7098b4cc7f1e04d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43299 } } peers { permanent_uuid: "794a373e8fa545c0bb05aa47f0abb056" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35739 } }
I20260618 22:43:18.686679 24132 heartbeater.cc:503] Master 127.5.213.254:44029 was elected leader, sending a full tablet report...
I20260618 22:43:18.686359 24152 raft_consensus.cc:348] T d23403f3d3c4443087bc3d3a91e7593d P 794a373e8fa545c0bb05aa47f0abb056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9aa07cd2f8544a28ccd6a563c89caba" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35085 } } peers { permanent_uuid: "4bcff14675d247fcb7098b4cc7f1e04d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43299 } } peers { permanent_uuid: "794a373e8fa545c0bb05aa47f0abb056" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35739 } }
I20260618 22:43:18.686861 24152 raft_consensus.cc:374] T d23403f3d3c4443087bc3d3a91e7593d P 794a373e8fa545c0bb05aa47f0abb056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:18.686951 24152 raft_consensus.cc:729] T d23403f3d3c4443087bc3d3a91e7593d P 794a373e8fa545c0bb05aa47f0abb056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 794a373e8fa545c0bb05aa47f0abb056, State: Initialized, Role: FOLLOWER
I20260618 22:43:18.687317 24151 ts_tablet_manager.cc:1428] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
I20260618 22:43:18.687603 24152 consensus_queue.cc:260] T d23403f3d3c4443087bc3d3a91e7593d P 794a373e8fa545c0bb05aa47f0abb056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9aa07cd2f8544a28ccd6a563c89caba" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35085 } } peers { permanent_uuid: "4bcff14675d247fcb7098b4cc7f1e04d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43299 } } peers { permanent_uuid: "794a373e8fa545c0bb05aa47f0abb056" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35739 } }
I20260618 22:43:18.688119 24156 raft_consensus.cc:482] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:43:18.688385 24156 raft_consensus.cc:504] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9aa07cd2f8544a28ccd6a563c89caba" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35085 } } peers { permanent_uuid: "4bcff14675d247fcb7098b4cc7f1e04d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43299 } } peers { permanent_uuid: "794a373e8fa545c0bb05aa47f0abb056" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35739 } }
I20260618 22:43:18.690168 24156 leader_election.cc:288] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bcff14675d247fcb7098b4cc7f1e04d (127.5.213.195:43299), 794a373e8fa545c0bb05aa47f0abb056 (127.5.213.193:35739)
I20260618 22:43:18.696091 24152 ts_tablet_manager.cc:1428] T d23403f3d3c4443087bc3d3a91e7593d P 794a373e8fa545c0bb05aa47f0abb056: Time spent starting tablet: real 0.020s user 0.016s sys 0.003s
I20260618 22:43:18.696463 23871 heartbeater.cc:503] Master 127.5.213.254:44029 was elected leader, sending a full tablet report...
I20260618 22:43:18.701571 24148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.691263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:33459 (local address 127.5.213.195:43299)
0618 22:43:18.691453 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:18.691459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:18.695292 (+ 3833us) server_negotiation.cc:407] Connection header received
0618 22:43:18.695483 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:18.695489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:18.695568 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:18.695646 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:18.696419 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.697407 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.699649 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.700701 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.700819 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:18.700866 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:18.701213 (+ 347us) server_negotiation.cc:299] Negotiation successful
0618 22:43:18.701332 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 22:43:18.701565 24160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.690871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:43299 (local address 127.5.213.194:33459)
0618 22:43:18.695178 (+ 4307us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:18.695197 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:18.695336 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:18.695841 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:18.695846 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:18.695874 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:18.696236 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:18.696249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.697519 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.697524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:18.699437 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:18.699448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.700875 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.700881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:18.701045 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:18.701083 (+ 38us) client_negotiation.cc:769] Sending connection context
0618 22:43:18.701176 (+ 93us) client_negotiation.cc:240] Negotiation successful
0618 22:43:18.701271 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4146,"thread_start_us":3963,"threads_started":1}
I20260618 22:43:18.701759 24149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.691372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:43369 (local address 127.5.213.193:35739)
0618 22:43:18.691553 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:18.691558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:18.695914 (+ 4356us) server_negotiation.cc:407] Connection header received
0618 22:43:18.696217 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:18.696222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:18.696314 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:18.696393 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:18.697125 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.698048 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.700151 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.700910 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.701061 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:18.701133 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:18.701401 (+ 268us) server_negotiation.cc:299] Negotiation successful
0618 22:43:18.701542 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260618 22:43:18.702020 24159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.690941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35739 (local address 127.5.213.194:43369)
0618 22:43:18.695815 (+ 4874us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:18.695838 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:18.696038 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:18.696586 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:18.696591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:18.696613 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:18.696951 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:18.696963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.698186 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.698192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:18.699988 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:18.699999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.701065 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.701070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:18.701253 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:18.701285 (+ 32us) client_negotiation.cc:769] Sending connection context
0618 22:43:18.701367 (+ 82us) client_negotiation.cc:240] Negotiation successful
0618 22:43:18.701484 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4759,"mutex_wait_us":3,"thread_start_us":4607,"threads_started":1}
I20260618 22:43:18.702432 24087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d23403f3d3c4443087bc3d3a91e7593d" candidate_uuid: "d9aa07cd2f8544a28ccd6a563c89caba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcff14675d247fcb7098b4cc7f1e04d" is_pre_election: true
I20260618 22:43:18.702953 24087 raft_consensus.cc:2454] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9aa07cd2f8544a28ccd6a563c89caba in term 0.
I20260618 22:43:18.702978 23826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d23403f3d3c4443087bc3d3a91e7593d" candidate_uuid: "d9aa07cd2f8544a28ccd6a563c89caba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "794a373e8fa545c0bb05aa47f0abb056" is_pre_election: true
I20260618 22:43:18.703428 23826 raft_consensus.cc:2454] T d23403f3d3c4443087bc3d3a91e7593d P 794a373e8fa545c0bb05aa47f0abb056 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9aa07cd2f8544a28ccd6a563c89caba in term 0.
I20260618 22:43:18.703568 23891 leader_election.cc:302] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcff14675d247fcb7098b4cc7f1e04d, d9aa07cd2f8544a28ccd6a563c89caba; no voters:
I20260618 22:43:18.703950 24156 raft_consensus.cc:2781] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:43:18.704072 24156 raft_consensus.cc:482] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:43:18.704167 24156 raft_consensus.cc:3037] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:18.706278 24156 raft_consensus.cc:504] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9aa07cd2f8544a28ccd6a563c89caba" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35085 } } peers { permanent_uuid: "4bcff14675d247fcb7098b4cc7f1e04d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43299 } } peers { permanent_uuid: "794a373e8fa545c0bb05aa47f0abb056" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35739 } }
I20260618 22:43:18.706890 24156 leader_election.cc:288] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [CANDIDATE]: Term 1 election: Requested vote from peers 4bcff14675d247fcb7098b4cc7f1e04d (127.5.213.195:43299), 794a373e8fa545c0bb05aa47f0abb056 (127.5.213.193:35739)
I20260618 22:43:18.707250 24087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d23403f3d3c4443087bc3d3a91e7593d" candidate_uuid: "d9aa07cd2f8544a28ccd6a563c89caba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcff14675d247fcb7098b4cc7f1e04d"
I20260618 22:43:18.707451 24087 raft_consensus.cc:3037] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:18.707469 23826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d23403f3d3c4443087bc3d3a91e7593d" candidate_uuid: "d9aa07cd2f8544a28ccd6a563c89caba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "794a373e8fa545c0bb05aa47f0abb056"
I20260618 22:43:18.707664 23826 raft_consensus.cc:3037] T d23403f3d3c4443087bc3d3a91e7593d P 794a373e8fa545c0bb05aa47f0abb056 [term 0 FOLLOWER]: Advancing to term 1
W20260618 22:43:18.709848 23872 tablet.cc:2369] T d23403f3d3c4443087bc3d3a91e7593d P 794a373e8fa545c0bb05aa47f0abb056: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:43:18.710456 24087 raft_consensus.cc:2454] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9aa07cd2f8544a28ccd6a563c89caba in term 1.
I20260618 22:43:18.710848 23891 leader_election.cc:302] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcff14675d247fcb7098b4cc7f1e04d, d9aa07cd2f8544a28ccd6a563c89caba; no voters:
I20260618 22:43:18.711089 24156 raft_consensus.cc:2781] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:18.711549 24156 raft_consensus.cc:686] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [term 1 LEADER]: Becoming Leader. State: Replica: d9aa07cd2f8544a28ccd6a563c89caba, State: Running, Role: LEADER
I20260618 22:43:18.711568 23826 raft_consensus.cc:2454] T d23403f3d3c4443087bc3d3a91e7593d P 794a373e8fa545c0bb05aa47f0abb056 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9aa07cd2f8544a28ccd6a563c89caba in term 1.
I20260618 22:43:18.712060 24156 consensus_queue.cc:237] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9aa07cd2f8544a28ccd6a563c89caba" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35085 } } peers { permanent_uuid: "4bcff14675d247fcb7098b4cc7f1e04d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43299 } } peers { permanent_uuid: "794a373e8fa545c0bb05aa47f0abb056" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35739 } }
I20260618 22:43:18.716995 23684 catalog_manager.cc:5495] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba reported cstate change: term changed from 0 to 1, leader changed from <none> to d9aa07cd2f8544a28ccd6a563c89caba (127.5.213.194). New cstate: current_term: 1 leader_uuid: "d9aa07cd2f8544a28ccd6a563c89caba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9aa07cd2f8544a28ccd6a563c89caba" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 35085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bcff14675d247fcb7098b4cc7f1e04d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794a373e8fa545c0bb05aa47f0abb056" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35739 } health_report { overall_health: UNKNOWN } } }
I20260618 22:43:18.737217 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.732035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41898 (local address 127.5.213.194:35085)
0618 22:43:18.732227 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:18.732232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:18.732248 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:43:18.732348 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:18.732352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:18.732426 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:18.732543 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:43:18.732855 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.733562 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.735000 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.735452 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.735557 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:18.736046 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:43:18.736553 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:43:18.736713 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:18.736842 (+ 129us) server_negotiation.cc:299] Negotiation successful
0618 22:43:18.736927 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 22:43:18.745718 24087 raft_consensus.cc:1261] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d [term 1 FOLLOWER]: Refusing update from remote peer d9aa07cd2f8544a28ccd6a563c89caba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:43:18.745716 23826 raft_consensus.cc:1261] T d23403f3d3c4443087bc3d3a91e7593d P 794a373e8fa545c0bb05aa47f0abb056 [term 1 FOLLOWER]: Refusing update from remote peer d9aa07cd2f8544a28ccd6a563c89caba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:43:18.746886 24161 consensus_queue.cc:1035] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [LEADER]: Connected to new peer: Peer: permanent_uuid: "794a373e8fa545c0bb05aa47f0abb056" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 35739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:43:18.747233 24156 consensus_queue.cc:1035] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcff14675d247fcb7098b4cc7f1e04d" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 43299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:43:18.758421 24164 mvcc.cc:205] Tried to move back new op lower bound from 7298345364451463168 to 7298345364328366080. Current Snapshot: MvccSnapshot[applied={T|T < 7298345364451463168}]
I20260618 22:43:18.759840 24166 mvcc.cc:205] Tried to move back new op lower bound from 7298345364451463168 to 7298345364328366080. Current Snapshot: MvccSnapshot[applied={T|T < 7298345364451463168}]
I20260618 22:43:18.760146 24165 mvcc.cc:205] Tried to move back new op lower bound from 7298345364451463168 to 7298345364328366080. Current Snapshot: MvccSnapshot[applied={T|T < 7298345364451463168}]
I20260618 22:43:18.772030 23685 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:43:18.772300 23685 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:43:18.776021 23685 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c69abaf9d2f445c1b8d36f18d5a0f4a7: Sending DeleteTablet for 3 replicas of tablet d23403f3d3c4443087bc3d3a91e7593d
I20260618 22:43:18.777112 23937 tablet_service.cc:1514] Processing DeleteTablet for tablet d23403f3d3c4443087bc3d3a91e7593d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:43:18 UTC) from {username='slave'} at 127.0.0.1:41892
I20260618 22:43:18.777225 23806 tablet_service.cc:1514] Processing DeleteTablet for tablet d23403f3d3c4443087bc3d3a91e7593d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:43:18 UTC) from {username='slave'} at 127.0.0.1:53058
I20260618 22:43:18.777217 24067 tablet_service.cc:1514] Processing DeleteTablet for tablet d23403f3d3c4443087bc3d3a91e7593d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:43:18 UTC) from {username='slave'} at 127.0.0.1:58052
I20260618 22:43:18.777875 24174 tablet_replica.cc:330] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba: stopping tablet replica
I20260618 22:43:18.778267 24174 raft_consensus.cc:2229] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:43:18.778956 24174 raft_consensus.cc:2258] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:43:18.779280 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 23744
I20260618 22:43:18.788681 24175 tablet_replica.cc:330] T d23403f3d3c4443087bc3d3a91e7593d P 794a373e8fa545c0bb05aa47f0abb056: stopping tablet replica
I20260618 22:43:18.788965 24174 ts_tablet_manager.cc:1905] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:43:18.789425 24175 raft_consensus.cc:2229] T d23403f3d3c4443087bc3d3a91e7593d P 794a373e8fa545c0bb05aa47f0abb056 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:43:18.790024 24175 raft_consensus.cc:2258] T d23403f3d3c4443087bc3d3a91e7593d P 794a373e8fa545c0bb05aa47f0abb056 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:43:18.790652 24176 tablet_replica.cc:330] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d: stopping tablet replica
I20260618 22:43:18.791558 24176 raft_consensus.cc:2229] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:43:18.792992 24175 ts_tablet_manager.cc:1905] T d23403f3d3c4443087bc3d3a91e7593d P 794a373e8fa545c0bb05aa47f0abb056: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:43:18.793850 24176 raft_consensus.cc:2258] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:43:18.795893 24149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.780165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53062 (local address 127.5.213.193:35739)
0618 22:43:18.780329 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:18.780336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:18.787098 (+ 6762us) server_negotiation.cc:407] Connection header received
0618 22:43:18.787282 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:18.787288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:18.787401 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:18.787496 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:18.788207 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.789416 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.791706 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.792350 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.792440 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:18.793795 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:18.793805 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:18.793822 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:18.794517 (+ 695us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:18.794575 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:18.794603 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:18.794731 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:18.795404 (+ 673us) server_negotiation.cc:299] Negotiation successful
0618 22:43:18.795519 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:43:18.796054 24174 ts_tablet_manager.cc:1918] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:43:18.796226 24174 log.cc:1196] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/d23403f3d3c4443087bc3d3a91e7593d
I20260618 22:43:18.796818 24174 ts_tablet_manager.cc:1939] T d23403f3d3c4443087bc3d3a91e7593d P d9aa07cd2f8544a28ccd6a563c89caba: Deleting consensus metadata
I20260618 22:43:18.797713 24176 ts_tablet_manager.cc:1905] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:43:18.798483 23672 catalog_manager.cc:4841] TS d9aa07cd2f8544a28ccd6a563c89caba (127.5.213.194:35085): tablet d23403f3d3c4443087bc3d3a91e7593d (table test-table [id=b585e41fc18440c588fa6ee9737ad074]) successfully deleted
I20260618 22:43:18.799016 23866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:18.805226 24176 ts_tablet_manager.cc:1918] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:43:18.805398 24176 log.cc:1196] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/d23403f3d3c4443087bc3d3a91e7593d
I20260618 22:43:18.805912 24176 ts_tablet_manager.cc:1939] T d23403f3d3c4443087bc3d3a91e7593d P 4bcff14675d247fcb7098b4cc7f1e04d: Deleting consensus metadata
I20260618 22:43:18.807807 23670 catalog_manager.cc:4841] TS 4bcff14675d247fcb7098b4cc7f1e04d (127.5.213.195:43299): tablet d23403f3d3c4443087bc3d3a91e7593d (table test-table [id=b585e41fc18440c588fa6ee9737ad074]) successfully deleted
I20260618 22:43:18.886251 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 23744
W20260618 22:43:18.907608 23671 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.213.193:35739 (error 108)
I20260618 22:43:18.908336 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 23875
I20260618 22:43:18.909636 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.909166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35739 (local address 127.0.0.1:53066)
0618 22:43:18.909345 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:18.909470 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260618 22:43:18.909857 23671 catalog_manager.cc:4584] TS 794a373e8fa545c0bb05aa47f0abb056 (127.5.213.193:35739): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d23403f3d3c4443087bc3d3a91e7593d: Network error: Client connection negotiation failed: client connection to 127.5.213.193:35739: connect: Connection refused (error 111)
I20260618 22:43:18.916733 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.909269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.5.213.194:35085)
0618 22:43:18.909440 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:18.909444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:18.910118 (+ 674us) server_negotiation.cc:407] Connection header received
0618 22:43:18.910256 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:18.910260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:18.910345 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:18.910426 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:18.910907 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.911651 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.913466 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:18.913900 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:18.913995 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:18.915292 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:18.915306 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:18.915320 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:18.915934 (+ 614us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:18.915992 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:18.916013 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:18.916099 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:18.916260 (+ 161us) server_negotiation.cc:299] Negotiation successful
0618 22:43:18.916375 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:43:18.917516 23997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:18.963671 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.963121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35739 (local address 127.0.0.1:53072)
0618 22:43:18.963360 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:18.963473 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260618 22:43:18.964645 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:18.964283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35739 (local address 127.0.0.1:53080)
0618 22:43:18.964425 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:18.964507 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260618 22:43:19.025374 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:19.024843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35739 (local address 127.0.0.1:53090)
0618 22:43:19.025067 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:19.025178 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260618 22:43:19.025942 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 23875
I20260618 22:43:19.026558 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:19.026100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35739 (local address 127.0.0.1:53092)
0618 22:43:19.026318 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:19.026402 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260618 22:43:19.041836 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 24005
I20260618 22:43:19.048362 24148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:19.042516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.5.213.195:43299)
0618 22:43:19.042784 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:19.042789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:19.042812 (+ 23us) server_negotiation.cc:407] Connection header received
0618 22:43:19.042941 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:19.042944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:19.043027 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:19.043109 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:19.043534 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:19.044252 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:19.045531 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:19.045934 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:19.046040 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:19.047048 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:19.047058 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:19.047072 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:19.047650 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:19.047698 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:19.047727 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:19.047826 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:19.047944 (+ 118us) server_negotiation.cc:299] Negotiation successful
0618 22:43:19.048054 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260618 22:43:19.049010 24127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:19.107079 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:19.106584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35739 (local address 127.0.0.1:53108)
0618 22:43:19.106796 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:19.106920 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260618 22:43:19.108137 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:19.107734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:35739 (local address 127.0.0.1:53124)
0618 22:43:19.107958 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:19.108028 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260618 22:43:19.134943 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 24005
I20260618 22:43:19.148092 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 23654
I20260618 22:43:19.149137 23715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:19.255450 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 23654
2026-06-18T22:43:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2575 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:43:19 dist-test-slave-lnj3 krb5kdc[24185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:43:19 dist-test-slave-lnj3 krb5kdc[24185](info): set up 2 sockets
Jun 18 22:43:19 dist-test-slave-lnj3 krb5kdc[24185](info): commencing operation
krb5kdc: starting...
W20260618 22:43:20.278584 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:43:20 dist-test-slave-lnj3 krb5kdc[24185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:43:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:43:20.513593 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:36909
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:36909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:20.837325 24198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:20.837666 24198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:20.837813 24198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:20.845343 24198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:43:20.845472 24198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:20.845554 24198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:20.845639 24198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:43:20.845702 24198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:43:20.855947 24198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:36909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:36909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:20.857285 24198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:20.858593 24198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:20.870118 24204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:20.870155 24206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:20.870074 24203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:20.870950 24198 server_base.cc:1034] running on GCE node
I20260618 22:43:20.871752 24198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:20.873229 24198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:20.874476 24198 hybrid_clock.cc:648] HybridClock initialized: now 1781822600874405 us; error 53 us; skew 500 ppm
I20260618 22:43:20.874943 24198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:20 dist-test-slave-lnj3 krb5kdc[24185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822600, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:20.878299 24198 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:43:20.879941 24198 webserver.cc:460] Webserver started at http://127.5.213.254:43669/ using document root <none> and password file <none>
I20260618 22:43:20.880630 24198 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:20.880743 24198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:20.881017 24198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:20.884016 24198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "abdf8240f20142318b3fec3060cf2841"
format_stamp: "Formatted at 2026-06-18 22:43:20 on dist-test-slave-lnj3"
server_key: "606a228e00a0ac768bd38beeb3540bb3"
server_key_iv: "b756273ceabd29afe85c7d65db7ad7ee"
server_key_version: "encryptionkey@0"
I20260618 22:43:20.884720 24198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "abdf8240f20142318b3fec3060cf2841"
format_stamp: "Formatted at 2026-06-18 22:43:20 on dist-test-slave-lnj3"
server_key: "606a228e00a0ac768bd38beeb3540bb3"
server_key_iv: "b756273ceabd29afe85c7d65db7ad7ee"
server_key_version: "encryptionkey@0"
I20260618 22:43:20.889696 24198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260618 22:43:20.893221 24213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:20.894757 24198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:43:20.894927 24198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "abdf8240f20142318b3fec3060cf2841"
format_stamp: "Formatted at 2026-06-18 22:43:20 on dist-test-slave-lnj3"
server_key: "606a228e00a0ac768bd38beeb3540bb3"
server_key_iv: "b756273ceabd29afe85c7d65db7ad7ee"
server_key_version: "encryptionkey@0"
I20260618 22:43:20.895129 24198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:20.929502 24198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:20.934230 24198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:20.934525 24198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:20.952484 24198 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:36909
I20260618 22:43:20.952509 24264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:36909 every 8 connection(s)
I20260618 22:43:20.953902 24198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:43:20.957543 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 24198
I20260618 22:43:20.957706 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:43:20.957805 24265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:20.958169 5975 external_mini_cluster.cc:1428] Setting key 4a4008a42a8a865ca1f9a1c4997e2199
I20260618 22:43:20.971830 24265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841: Bootstrap starting.
Jun 18 22:43:20 dist-test-slave-lnj3 krb5kdc[24185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:43:20.975386 24265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:20.976460 24265 log.cc:824] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:20.979631 24265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841: No bootstrap required, opened a new log
I20260618 22:43:20.980602 24268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:20.959884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39232 (local address 127.5.213.254:36909)
0618 22:43:20.960674 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:20.960685 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:20.960728 (+ 43us) server_negotiation.cc:407] Connection header received
0618 22:43:20.962415 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:20.962436 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:20.962875 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:20.963198 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:20.965001 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:20.966240 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:20.968083 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:20.968797 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:20.968928 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:20.973740 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:20.973748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:20.973762 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:20.973799 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:20.976725 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:20.977441 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:20.977449 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:20.977457 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:20.977654 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:20.978070 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:20.978073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:20.978075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:20.978580 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:20.978841 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:20.979211 (+ 370us) server_negotiation.cc:299] Negotiation successful
0618 22:43:20.979564 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":367,"threads_started":1}
I20260618 22:43:20.987023 24265 raft_consensus.cc:348] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdf8240f20142318b3fec3060cf2841" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36909 } }
I20260618 22:43:20.987401 24265 raft_consensus.cc:374] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:20.987562 24265 raft_consensus.cc:729] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abdf8240f20142318b3fec3060cf2841, State: Initialized, Role: FOLLOWER
I20260618 22:43:20.988131 24265 consensus_queue.cc:260] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdf8240f20142318b3fec3060cf2841" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36909 } }
I20260618 22:43:20.988353 24265 raft_consensus.cc:388] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:43:20.988477 24265 raft_consensus.cc:482] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:43:20.988624 24265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:20.990805 24265 raft_consensus.cc:504] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdf8240f20142318b3fec3060cf2841" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36909 } }
I20260618 22:43:20.991297 24265 leader_election.cc:302] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abdf8240f20142318b3fec3060cf2841; no voters:
I20260618 22:43:20.991860 24265 leader_election.cc:288] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:43:20.992027 24270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:20.992648 24270 raft_consensus.cc:686] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841 [term 1 LEADER]: Becoming Leader. State: Replica: abdf8240f20142318b3fec3060cf2841, State: Running, Role: LEADER
I20260618 22:43:20.993182 24270 consensus_queue.cc:237] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdf8240f20142318b3fec3060cf2841" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36909 } }
I20260618 22:43:20.993777 24265 sys_catalog.cc:564] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:43:20.996030 24271 sys_catalog.cc:455] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abdf8240f20142318b3fec3060cf2841" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdf8240f20142318b3fec3060cf2841" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36909 } } }
I20260618 22:43:20.996829 24271 sys_catalog.cc:458] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:20.996220 24272 sys_catalog.cc:455] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abdf8240f20142318b3fec3060cf2841. Latest consensus state: current_term: 1 leader_uuid: "abdf8240f20142318b3fec3060cf2841" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdf8240f20142318b3fec3060cf2841" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 36909 } } }
I20260618 22:43:20.997525 24272 sys_catalog.cc:458] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:21.005798 24277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:43:21.011112 24277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:43:21.019855 24277 catalog_manager.cc:1347] Generated new cluster ID: f230a1d8a180473394fdbf254b9fc9e6
I20260618 22:43:21.019961 24277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:43:21.046356 24277 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:43:21.048259 24277 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:43:21.055857 24277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841: Generated new TSK 0
I20260618 22:43:21.056902 24277 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
I20260618 22:43:21.119726 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36909
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:21.439736 24291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:21.440092 24291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:21.440222 24291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:21.447749 24291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:21.447857 24291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:21.448004 24291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:43:21.457924 24291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:21.459297 24291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:21.460660 24291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:21.475178 24297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:21.475114 24296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:21.475637 24299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:21.476797 24291 server_base.cc:1034] running on GCE node
I20260618 22:43:21.477461 24291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:21.478477 24291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:21.479696 24291 hybrid_clock.cc:648] HybridClock initialized: now 1781822601479611 us; error 70 us; skew 500 ppm
I20260618 22:43:21.480036 24291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:21 dist-test-slave-lnj3 krb5kdc[24185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822601, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:21.483361 24291 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:43:21.484905 24291 webserver.cc:460] Webserver started at http://127.5.213.193:37983/ using document root <none> and password file <none>
I20260618 22:43:21.485605 24291 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:21.485723 24291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:21.486027 24291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:21.489112 24291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "af9a152f4a8c4188ba8fb988fbfd7f5f"
format_stamp: "Formatted at 2026-06-18 22:43:21 on dist-test-slave-lnj3"
server_key: "b4f8e4a815fb422b1e8c9e309f3554a9"
server_key_iv: "e5b0fe444c7311ee6f8827c95ffd6d5b"
server_key_version: "encryptionkey@0"
I20260618 22:43:21.489966 24291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "af9a152f4a8c4188ba8fb988fbfd7f5f"
format_stamp: "Formatted at 2026-06-18 22:43:21 on dist-test-slave-lnj3"
server_key: "b4f8e4a815fb422b1e8c9e309f3554a9"
server_key_iv: "e5b0fe444c7311ee6f8827c95ffd6d5b"
server_key_version: "encryptionkey@0"
I20260618 22:43:21.495647 24291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260618 22:43:21.499413 24306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:21.501005 24291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:43:21.501173 24291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "af9a152f4a8c4188ba8fb988fbfd7f5f"
format_stamp: "Formatted at 2026-06-18 22:43:21 on dist-test-slave-lnj3"
server_key: "b4f8e4a815fb422b1e8c9e309f3554a9"
server_key_iv: "e5b0fe444c7311ee6f8827c95ffd6d5b"
server_key_version: "encryptionkey@0"
I20260618 22:43:21.501371 24291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:21.528240 24291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:21.532660 24291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:21.532945 24291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:21.533938 24291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:43:21.535359 24291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:43:21.535439 24291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:21.535563 24291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:43:21.535616 24291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:21.566772 24291 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:41277
I20260618 22:43:21.566867 24418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:41277 every 8 connection(s)
I20260618 22:43:21.568107 24291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:43:21.572458 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 24291
I20260618 22:43:21.572633 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:43:21.573074 5975 external_mini_cluster.cc:1428] Setting key 9ed2ce823fd1680134a6b41ab51f7e83
Jun 18 22:43:21 dist-test-slave-lnj3 krb5kdc[24185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822601, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:43:21.589309 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:21.570847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:42953 (local address 127.5.213.254:36909)
0618 22:43:21.571324 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:21.571330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:21.572444 (+ 1114us) server_negotiation.cc:407] Connection header received
0618 22:43:21.574593 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:21.574598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:21.574731 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:21.574833 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:21.576674 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:21.577699 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:21.579449 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:21.580021 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:21.580149 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:21.583753 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:21.583758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:21.583761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:21.583795 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:21.586221 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:21.587097 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:21.587101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:21.587103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:21.587238 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:21.587658 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:21.587661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:21.587663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:21.587912 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:21.588015 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:21.588855 (+ 840us) server_negotiation.cc:299] Negotiation successful
0618 22:43:21.589037 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":213,"threads_started":1}
I20260618 22:43:21.590770 24422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:21.571491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:36909 (local address 127.5.213.193:42953)
0618 22:43:21.572261 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:21.572303 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:21.574245 (+ 1942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:21.575157 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:21.575167 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:21.575804 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:43:21.576486 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:21.576508 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:21.577887 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:21.577891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:21.579309 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:21.579323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:21.580261 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:21.580266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:21.580478 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:21.581205 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:43:21.581231 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:43:21.583501 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:43:21.586367 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:43:21.586377 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:43:21.586396 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:43:21.586996 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:43:21.587325 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:43:21.587330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:43:21.587333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:43:21.587557 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:43:21.588026 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:43:21.588033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:43:21.588471 (+ 438us) client_negotiation.cc:769] Sending connection context
0618 22:43:21.588784 (+ 313us) client_negotiation.cc:240] Negotiation successful
0618 22:43:21.589318 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":298,"threads_started":1}
I20260618 22:43:21.592712 24419 heartbeater.cc:347] Connected to a master server at 127.5.213.254:36909
I20260618 22:43:21.593027 24419 heartbeater.cc:464] Registering TS with master...
I20260618 22:43:21.593830 24419 heartbeater.cc:511] Master 127.5.213.254:36909 requested a full tablet report, sending...
I20260618 22:43:21.596447 24230 ts_manager.cc:194] Registered new tserver with Master: af9a152f4a8c4188ba8fb988fbfd7f5f (127.5.213.193:41277)
WARNING: no policy specified for kudu/127.5.213.194@KRBTEST.COM; defaulting to no policy
I20260618 22:43:21.598371 24230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:42953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
I20260618 22:43:21.625950 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36909
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:21.948680 24425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:21.948961 24425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:21.949055 24425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:21.956645 24425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:21.956776 24425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:21.956923 24425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:43:21.966949 24425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:21.968256 24425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:21.969610 24425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:21.982488 24430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:21.990893 24433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:21.990893 24431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:21.991773 24425 server_base.cc:1034] running on GCE node
I20260618 22:43:21.992528 24425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:21.993628 24425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:21.994867 24425 hybrid_clock.cc:648] HybridClock initialized: now 1781822601994778 us; error 62 us; skew 500 ppm
I20260618 22:43:21.995245 24425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:21 dist-test-slave-lnj3 krb5kdc[24185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822601, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:21.998468 24425 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:43:22.000052 24425 webserver.cc:460] Webserver started at http://127.5.213.194:42065/ using document root <none> and password file <none>
I20260618 22:43:22.000784 24425 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:22.000900 24425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:22.001200 24425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:22.004328 24425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "5691534cfe3f40509e88203fe41ab1a3"
format_stamp: "Formatted at 2026-06-18 22:43:21 on dist-test-slave-lnj3"
server_key: "4be8bbe8497f49230f615c7a6acede41"
server_key_iv: "8f5116750194ed948353ce9117da4ad6"
server_key_version: "encryptionkey@0"
I20260618 22:43:22.005088 24425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "5691534cfe3f40509e88203fe41ab1a3"
format_stamp: "Formatted at 2026-06-18 22:43:21 on dist-test-slave-lnj3"
server_key: "4be8bbe8497f49230f615c7a6acede41"
server_key_iv: "8f5116750194ed948353ce9117da4ad6"
server_key_version: "encryptionkey@0"
I20260618 22:43:22.010556 24425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260618 22:43:22.014076 24440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:22.015450 24425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:43:22.015601 24425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "5691534cfe3f40509e88203fe41ab1a3"
format_stamp: "Formatted at 2026-06-18 22:43:21 on dist-test-slave-lnj3"
server_key: "4be8bbe8497f49230f615c7a6acede41"
server_key_iv: "8f5116750194ed948353ce9117da4ad6"
server_key_version: "encryptionkey@0"
I20260618 22:43:22.015780 24425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:22.037452 24425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:22.041625 24425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:22.041908 24425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:22.042912 24425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:43:22.044251 24425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:43:22.044328 24425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:22.044445 24425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:43:22.044497 24425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:22.073884 24425 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:34453
I20260618 22:43:22.073926 24552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:34453 every 8 connection(s)
I20260618 22:43:22.075202 24425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:43:22.081053 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 24425
I20260618 22:43:22.081239 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:43:22.081763 5975 external_mini_cluster.cc:1428] Setting key 61c291c263556309254b765040e4f46b
Jun 18 22:43:22 dist-test-slave-lnj3 krb5kdc[24185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822601, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:43:22.095845 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:22.077691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:54291 (local address 127.5.213.254:36909)
0618 22:43:22.077890 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:22.077896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:22.079107 (+ 1211us) server_negotiation.cc:407] Connection header received
0618 22:43:22.081004 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:22.081008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:22.081082 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:22.081176 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:22.083069 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.083787 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.085300 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.085733 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.085841 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:22.089377 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:22.089382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:22.089385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:22.089430 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:22.092481 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:22.093233 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:22.093238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:22.093241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:22.093384 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:22.093919 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:22.093923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:22.093925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:22.094261 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:22.094403 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:22.095246 (+ 843us) server_negotiation.cc:299] Negotiation successful
0618 22:43:22.095496 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 22:43:22.096909 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:22.078209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:36909 (local address 127.5.213.194:54291)
0618 22:43:22.078940 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:22.078987 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:22.080674 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:22.081451 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:22.081463 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:22.082216 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:43:22.082899 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:22.082919 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.083932 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.083936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:22.085164 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:22.085176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.085943 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.085951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:22.086167 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:22.086805 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:43:22.086827 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:43:22.089129 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:43:22.092638 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:43:22.092645 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:43:22.092660 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:43:22.093111 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:43:22.093535 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:43:22.093540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:43:22.093543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:43:22.093783 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:43:22.094390 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:43:22.094398 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:43:22.094826 (+ 428us) client_negotiation.cc:769] Sending connection context
0618 22:43:22.095128 (+ 302us) client_negotiation.cc:240] Negotiation successful
0618 22:43:22.095717 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":254,"threads_started":1}
I20260618 22:43:22.098537 24553 heartbeater.cc:347] Connected to a master server at 127.5.213.254:36909
I20260618 22:43:22.098958 24553 heartbeater.cc:464] Registering TS with master...
I20260618 22:43:22.100014 24553 heartbeater.cc:511] Master 127.5.213.254:36909 requested a full tablet report, sending...
I20260618 22:43:22.101783 24230 ts_manager.cc:194] Registered new tserver with Master: 5691534cfe3f40509e88203fe41ab1a3 (127.5.213.194:34453)
I20260618 22:43:22.102622 24230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:54291
WARNING: no policy specified for kudu/127.5.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
I20260618 22:43:22.133822 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36909
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:22.450314 24558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:22.450596 24558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:22.450757 24558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:22.458070 24558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:22.458184 24558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:22.458294 24558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:43:22.468209 24558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:36909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:22.469414 24558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:22.470737 24558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:22.483513 24563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:22.483438 24566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:22.483474 24564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:22.484670 24558 server_base.cc:1034] running on GCE node
I20260618 22:43:22.485271 24558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:22.486202 24558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:22.487406 24558 hybrid_clock.cc:648] HybridClock initialized: now 1781822602487351 us; error 44 us; skew 500 ppm
I20260618 22:43:22.487738 24558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:22 dist-test-slave-lnj3 krb5kdc[24185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822602, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:22.490901 24558 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:43:22.492352 24558 webserver.cc:460] Webserver started at http://127.5.213.195:34353/ using document root <none> and password file <none>
I20260618 22:43:22.493019 24558 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:22.493134 24558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:22.493451 24558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:22.496737 24558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "6adb703fabc14ddfae16d6c57d6f5697"
format_stamp: "Formatted at 2026-06-18 22:43:22 on dist-test-slave-lnj3"
server_key: "dca468e672eef54a07f2d6d26c8c06e8"
server_key_iv: "09ee164af66542bba340f713bba55a95"
server_key_version: "encryptionkey@0"
I20260618 22:43:22.497565 24558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "6adb703fabc14ddfae16d6c57d6f5697"
format_stamp: "Formatted at 2026-06-18 22:43:22 on dist-test-slave-lnj3"
server_key: "dca468e672eef54a07f2d6d26c8c06e8"
server_key_iv: "09ee164af66542bba340f713bba55a95"
server_key_version: "encryptionkey@0"
I20260618 22:43:22.503252 24558 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260618 22:43:22.507190 24573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:22.508605 24558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260618 22:43:22.508767 24558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "6adb703fabc14ddfae16d6c57d6f5697"
format_stamp: "Formatted at 2026-06-18 22:43:22 on dist-test-slave-lnj3"
server_key: "dca468e672eef54a07f2d6d26c8c06e8"
server_key_iv: "09ee164af66542bba340f713bba55a95"
server_key_version: "encryptionkey@0"
I20260618 22:43:22.508958 24558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:22.526361 24558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:22.531020 24558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:22.531301 24558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:22.532300 24558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:43:22.533883 24558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:43:22.533973 24558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:22.534098 24558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:43:22.534152 24558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:22.564416 24558 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:33599
I20260618 22:43:22.564457 24685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:33599 every 8 connection(s)
I20260618 22:43:22.565702 24558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:43:22.575863 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 24558
I20260618 22:43:22.576038 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:43:22.576617 5975 external_mini_cluster.cc:1428] Setting key f68e42cc58c4df602dd8fcf846a62cc2
Jun 18 22:43:22 dist-test-slave-lnj3 krb5kdc[24185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822602, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:43:22.586212 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:22.568342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:42665 (local address 127.5.213.254:36909)
0618 22:43:22.568546 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:22.568553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:22.569795 (+ 1242us) server_negotiation.cc:407] Connection header received
0618 22:43:22.572221 (+ 2426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:22.572227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:22.572324 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:22.572414 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:22.573857 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.574613 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.576224 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.576938 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.577076 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:22.580748 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:22.580753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:22.580757 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:22.580804 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:22.583325 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:22.584113 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:22.584117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:22.584120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:22.584272 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:22.584695 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:22.584699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:22.584701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:22.584992 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:22.585108 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:22.585778 (+ 670us) server_negotiation.cc:299] Negotiation successful
0618 22:43:22.585955 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260618 22:43:22.587283 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:22.568796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:36909 (local address 127.5.213.195:42665)
0618 22:43:22.569609 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:22.569653 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:22.571936 (+ 2283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:22.572611 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:22.572619 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:22.573142 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:43:22.573695 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:22.573713 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.574785 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.574789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:22.576039 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:22.576053 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.577160 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.577168 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:22.577438 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:22.578167 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:43:22.578191 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:43:22.580545 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:43:22.583469 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:43:22.583476 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:43:22.583489 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:43:22.583942 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:43:22.584376 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:43:22.584379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:43:22.584382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:43:22.584571 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:43:22.585108 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:43:22.585115 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:43:22.585453 (+ 338us) client_negotiation.cc:769] Sending connection context
0618 22:43:22.585692 (+ 239us) client_negotiation.cc:240] Negotiation successful
0618 22:43:22.586095 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":298,"threads_started":1}
I20260618 22:43:22.588615 24686 heartbeater.cc:347] Connected to a master server at 127.5.213.254:36909
I20260618 22:43:22.588902 24686 heartbeater.cc:464] Registering TS with master...
I20260618 22:43:22.589609 24686 heartbeater.cc:511] Master 127.5.213.254:36909 requested a full tablet report, sending...
I20260618 22:43:22.591035 24230 ts_manager.cc:194] Registered new tserver with Master: 6adb703fabc14ddfae16d6c57d6f5697 (127.5.213.195:33599)
I20260618 22:43:22.591822 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:43:22.591863 24230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:42665
I20260618 22:43:22.601774 24419 heartbeater.cc:503] Master 127.5.213.254:36909 was elected leader, sending a full tablet report...
I20260618 22:43:22.609551 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:22.595992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.5.213.254:36909)
0618 22:43:22.596232 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:22.596239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:22.596580 (+ 341us) server_negotiation.cc:407] Connection header received
0618 22:43:22.596720 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:22.596724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:22.596788 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:22.596876 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:22.598562 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.599355 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.600820 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.601240 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.601350 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:22.604047 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:22.604051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:22.604055 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:22.604088 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:22.606923 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:22.607634 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:22.607638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:22.607640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:22.607763 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:22.608130 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:22.608133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:22.608135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:22.608454 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:22.608652 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:22.609085 (+ 433us) server_negotiation.cc:299] Negotiation successful
0618 22:43:22.609236 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260618 22:43:22.613605 24230 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:43:22.620488 24230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:43:22.643796 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:22.633813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.5.213.193:41277)
0618 22:43:22.636547 (+ 2734us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:22.636554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:22.636581 (+ 27us) server_negotiation.cc:407] Connection header received
0618 22:43:22.636683 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:22.636690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:22.636940 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:22.637133 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:22.637955 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.639458 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.641137 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.642241 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.642397 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:22.642677 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:22.643275 (+ 598us) server_negotiation.cc:299] Negotiation successful
0618 22:43:22.643445 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2572,"thread_start_us":2401,"threads_started":1}
I20260618 22:43:22.643828 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:22.632655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:33599 (local address 127.0.0.1:42996)
0618 22:43:22.633496 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:22.633533 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:22.633696 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:22.636047 (+ 2351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:22.636055 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:22.636084 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:22.636556 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:22.636571 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.638719 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.638724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:22.641090 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:22.641104 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.642919 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.642925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:22.643098 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:22.643136 (+ 38us) client_negotiation.cc:769] Sending connection context
0618 22:43:22.643217 (+ 81us) client_negotiation.cc:240] Negotiation successful
0618 22:43:22.643445 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":21,"thread_start_us":2450,"threads_started":1}
I20260618 22:43:22.644121 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:22.633057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42996 (local address 127.5.213.195:33599)
0618 22:43:22.635406 (+ 2349us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:22.635414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:22.635444 (+ 30us) server_negotiation.cc:407] Connection header received
0618 22:43:22.635538 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:22.635545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:22.635804 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:22.636752 (+ 948us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:22.637012 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.638554 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.641286 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.642693 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.643373 (+ 680us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:22.643597 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:22.643660 (+ 63us) server_negotiation.cc:299] Negotiation successful
0618 22:43:22.643815 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2184,"thread_start_us":2028,"threads_started":1}
I20260618 22:43:22.644435 24701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:22.632758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:41277 (local address 127.0.0.1:52180)
0618 22:43:22.634764 (+ 2006us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:22.634780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:22.634878 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:22.637145 (+ 2267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:22.637151 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:22.637181 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:22.637802 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:22.637815 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.639571 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.639585 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:22.640979 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:22.640992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.642803 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.642809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:22.643023 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:22.643091 (+ 68us) client_negotiation.cc:769] Sending connection context
0618 22:43:22.643218 (+ 127us) client_negotiation.cc:240] Negotiation successful
0618 22:43:22.644070 (+ 852us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1899,"mutex_wait_us":3,"thread_start_us":4855,"threads_started":1}
I20260618 22:43:22.645969 24702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:22.633667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.5.213.194:34453)
0618 22:43:22.634347 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:22.634357 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:22.638025 (+ 3668us) server_negotiation.cc:407] Connection header received
0618 22:43:22.638163 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:22.638171 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:22.638401 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:22.638573 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:22.639499 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.641017 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.643395 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.644984 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.645103 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:22.645295 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:22.645511 (+ 216us) server_negotiation.cc:299] Negotiation successful
0618 22:43:22.645685 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":392,"threads_started":1}
I20260618 22:43:22.646713 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:22.633138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:34453 (local address 127.0.0.1:49534)
0618 22:43:22.637900 (+ 4762us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:22.637918 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:22.638030 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:22.638861 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:22.638866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:22.638894 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:22.639290 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:22.639300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.641188 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.641192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:22.643239 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:22.643252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.645176 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.645183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:22.645342 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:22.645372 (+ 30us) client_negotiation.cc:769] Sending connection context
0618 22:43:22.645439 (+ 67us) client_negotiation.cc:240] Negotiation successful
0618 22:43:22.645553 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4628,"thread_start_us":1356,"threads_started":1}
I20260618 22:43:22.648350 24354 tablet_service.cc:1467] Processing CreateTablet for tablet 68c7023e105a4528bb9f569e9495ab85 (DEFAULT_TABLE table=test-table [id=c5b6ed9a549b474088303035d898c693]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:43:22.648144 24621 tablet_service.cc:1467] Processing CreateTablet for tablet 68c7023e105a4528bb9f569e9495ab85 (DEFAULT_TABLE table=test-table [id=c5b6ed9a549b474088303035d898c693]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:43:22.649528 24354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68c7023e105a4528bb9f569e9495ab85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:22.649621 24621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68c7023e105a4528bb9f569e9495ab85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:22.650067 24488 tablet_service.cc:1467] Processing CreateTablet for tablet 68c7023e105a4528bb9f569e9495ab85 (DEFAULT_TABLE table=test-table [id=c5b6ed9a549b474088303035d898c693]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:43:22.651383 24488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68c7023e105a4528bb9f569e9495ab85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:22.660658 24704 tablet_bootstrap.cc:492] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f: Bootstrap starting.
I20260618 22:43:22.663568 24705 tablet_bootstrap.cc:492] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697: Bootstrap starting.
I20260618 22:43:22.664367 24704 tablet_bootstrap.cc:654] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:22.665220 24706 tablet_bootstrap.cc:492] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3: Bootstrap starting.
I20260618 22:43:22.667061 24705 tablet_bootstrap.cc:654] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:22.667210 24704 log.cc:824] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:22.668256 24706 tablet_bootstrap.cc:654] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:22.668510 24705 log.cc:824] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:22.669365 24706 log.cc:824] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:22.670275 24704 tablet_bootstrap.cc:492] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f: No bootstrap required, opened a new log
I20260618 22:43:22.670555 24704 ts_tablet_manager.cc:1397] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260618 22:43:22.671582 24705 tablet_bootstrap.cc:492] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697: No bootstrap required, opened a new log
I20260618 22:43:22.671837 24705 ts_tablet_manager.cc:1397] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260618 22:43:22.672122 24706 tablet_bootstrap.cc:492] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3: No bootstrap required, opened a new log
I20260618 22:43:22.672334 24706 ts_tablet_manager.cc:1397] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260618 22:43:22.679952 24706 raft_consensus.cc:348] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb703fabc14ddfae16d6c57d6f5697" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33599 } } peers { permanent_uuid: "af9a152f4a8c4188ba8fb988fbfd7f5f" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 41277 } } peers { permanent_uuid: "5691534cfe3f40509e88203fe41ab1a3" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 34453 } }
I20260618 22:43:22.680358 24706 raft_consensus.cc:374] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:22.680455 24706 raft_consensus.cc:729] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5691534cfe3f40509e88203fe41ab1a3, State: Initialized, Role: FOLLOWER
I20260618 22:43:22.681107 24706 consensus_queue.cc:260] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb703fabc14ddfae16d6c57d6f5697" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33599 } } peers { permanent_uuid: "af9a152f4a8c4188ba8fb988fbfd7f5f" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 41277 } } peers { permanent_uuid: "5691534cfe3f40509e88203fe41ab1a3" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 34453 } }
I20260618 22:43:22.681609 24704 raft_consensus.cc:348] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb703fabc14ddfae16d6c57d6f5697" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33599 } } peers { permanent_uuid: "af9a152f4a8c4188ba8fb988fbfd7f5f" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 41277 } } peers { permanent_uuid: "5691534cfe3f40509e88203fe41ab1a3" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 34453 } }
I20260618 22:43:22.682091 24704 raft_consensus.cc:374] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:22.682186 24704 raft_consensus.cc:729] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9a152f4a8c4188ba8fb988fbfd7f5f, State: Initialized, Role: FOLLOWER
I20260618 22:43:22.682216 24553 heartbeater.cc:503] Master 127.5.213.254:36909 was elected leader, sending a full tablet report...
I20260618 22:43:22.682197 24705 raft_consensus.cc:348] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb703fabc14ddfae16d6c57d6f5697" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33599 } } peers { permanent_uuid: "af9a152f4a8c4188ba8fb988fbfd7f5f" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 41277 } } peers { permanent_uuid: "5691534cfe3f40509e88203fe41ab1a3" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 34453 } }
I20260618 22:43:22.682619 24705 raft_consensus.cc:374] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:22.682749 24705 raft_consensus.cc:729] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adb703fabc14ddfae16d6c57d6f5697, State: Initialized, Role: FOLLOWER
I20260618 22:43:22.682950 24706 ts_tablet_manager.cc:1428] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260618 22:43:22.682968 24704 consensus_queue.cc:260] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb703fabc14ddfae16d6c57d6f5697" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33599 } } peers { permanent_uuid: "af9a152f4a8c4188ba8fb988fbfd7f5f" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 41277 } } peers { permanent_uuid: "5691534cfe3f40509e88203fe41ab1a3" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 34453 } }
I20260618 22:43:22.683429 24705 consensus_queue.cc:260] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb703fabc14ddfae16d6c57d6f5697" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33599 } } peers { permanent_uuid: "af9a152f4a8c4188ba8fb988fbfd7f5f" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 41277 } } peers { permanent_uuid: "5691534cfe3f40509e88203fe41ab1a3" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 34453 } }
I20260618 22:43:22.685066 24686 heartbeater.cc:503] Master 127.5.213.254:36909 was elected leader, sending a full tablet report...
I20260618 22:43:22.685801 24705 ts_tablet_manager.cc:1428] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260618 22:43:22.687738 24704 ts_tablet_manager.cc:1428] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f: Time spent starting tablet: real 0.017s user 0.013s sys 0.000s
W20260618 22:43:22.817394 24687 tablet.cc:2369] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:43:22.820533 24420 tablet.cc:2369] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:43:22.827069 24554 tablet.cc:2369] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:43:22.928963 24711 raft_consensus.cc:482] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:43:22.929278 24711 raft_consensus.cc:504] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb703fabc14ddfae16d6c57d6f5697" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33599 } } peers { permanent_uuid: "af9a152f4a8c4188ba8fb988fbfd7f5f" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 41277 } } peers { permanent_uuid: "5691534cfe3f40509e88203fe41ab1a3" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 34453 } }
I20260618 22:43:22.931269 24711 leader_election.cc:288] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6adb703fabc14ddfae16d6c57d6f5697 (127.5.213.195:33599), 5691534cfe3f40509e88203fe41ab1a3 (127.5.213.194:34453)
I20260618 22:43:22.938438 24702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:22.931731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:59445 (local address 127.5.213.194:34453)
0618 22:43:22.932068 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:22.932081 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:22.932815 (+ 734us) server_negotiation.cc:407] Connection header received
0618 22:43:22.933225 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:22.933231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:22.933310 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:22.933386 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:22.934038 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.934819 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.936650 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.937472 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.937590 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:22.937646 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:22.938053 (+ 407us) server_negotiation.cc:299] Negotiation successful
0618 22:43:22.938196 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260618 22:43:22.938452 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:22.931824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:46399 (local address 127.5.213.195:33599)
0618 22:43:22.932049 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:22.932056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:22.933212 (+ 1156us) server_negotiation.cc:407] Connection header received
0618 22:43:22.933361 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:22.933366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:22.933444 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:22.933537 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:22.934145 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.935001 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.936822 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.937618 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.937787 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:22.937859 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:22.938086 (+ 227us) server_negotiation.cc:299] Negotiation successful
0618 22:43:22.938208 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260618 22:43:22.938513 24713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:22.931925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:34453 (local address 127.5.213.193:59445)
0618 22:43:22.932704 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:22.932725 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:22.933046 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:22.933439 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:22.933444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:22.933468 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:22.933886 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:22.933897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.934946 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.934952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:22.936489 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:22.936501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.937680 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.937687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:22.937893 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:22.937923 (+ 30us) client_negotiation.cc:769] Sending connection context
0618 22:43:22.937996 (+ 73us) client_negotiation.cc:240] Negotiation successful
0618 22:43:22.938127 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":623,"threads_started":1}
I20260618 22:43:22.938820 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:22.932323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:33599 (local address 127.5.213.193:46399)
0618 22:43:22.933095 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:22.933115 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:22.933240 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:22.933564 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:22.933567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:22.933608 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:22.933974 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:22.933985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.935144 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.935148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:22.936664 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:22.936678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:22.937820 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:22.937825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:22.937936 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:22.937958 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:43:22.938045 (+ 87us) client_negotiation.cc:240] Negotiation successful
0618 22:43:22.938169 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":2,"thread_start_us":478,"threads_started":1}
I20260618 22:43:22.939507 24508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68c7023e105a4528bb9f569e9495ab85" candidate_uuid: "af9a152f4a8c4188ba8fb988fbfd7f5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5691534cfe3f40509e88203fe41ab1a3" is_pre_election: true
I20260618 22:43:22.939851 24641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68c7023e105a4528bb9f569e9495ab85" candidate_uuid: "af9a152f4a8c4188ba8fb988fbfd7f5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb703fabc14ddfae16d6c57d6f5697" is_pre_election: true
I20260618 22:43:22.940155 24508 raft_consensus.cc:2454] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af9a152f4a8c4188ba8fb988fbfd7f5f in term 0.
I20260618 22:43:22.940366 24641 raft_consensus.cc:2454] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af9a152f4a8c4188ba8fb988fbfd7f5f in term 0.
I20260618 22:43:22.940896 24309 leader_election.cc:302] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5691534cfe3f40509e88203fe41ab1a3, af9a152f4a8c4188ba8fb988fbfd7f5f; no voters:
I20260618 22:43:22.941293 24711 raft_consensus.cc:2781] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:43:22.941418 24711 raft_consensus.cc:482] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:43:22.941511 24711 raft_consensus.cc:3037] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:22.943691 24711 raft_consensus.cc:504] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb703fabc14ddfae16d6c57d6f5697" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33599 } } peers { permanent_uuid: "af9a152f4a8c4188ba8fb988fbfd7f5f" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 41277 } } peers { permanent_uuid: "5691534cfe3f40509e88203fe41ab1a3" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 34453 } }
I20260618 22:43:22.944279 24711 leader_election.cc:288] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [CANDIDATE]: Term 1 election: Requested vote from peers 6adb703fabc14ddfae16d6c57d6f5697 (127.5.213.195:33599), 5691534cfe3f40509e88203fe41ab1a3 (127.5.213.194:34453)
I20260618 22:43:22.944661 24641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68c7023e105a4528bb9f569e9495ab85" candidate_uuid: "af9a152f4a8c4188ba8fb988fbfd7f5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adb703fabc14ddfae16d6c57d6f5697"
I20260618 22:43:22.944764 24508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68c7023e105a4528bb9f569e9495ab85" candidate_uuid: "af9a152f4a8c4188ba8fb988fbfd7f5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5691534cfe3f40509e88203fe41ab1a3"
I20260618 22:43:22.944826 24641 raft_consensus.cc:3037] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:22.944960 24508 raft_consensus.cc:3037] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:22.946966 24641 raft_consensus.cc:2454] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af9a152f4a8c4188ba8fb988fbfd7f5f in term 1.
I20260618 22:43:22.947211 24508 raft_consensus.cc:2454] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af9a152f4a8c4188ba8fb988fbfd7f5f in term 1.
I20260618 22:43:22.947331 24307 leader_election.cc:302] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6adb703fabc14ddfae16d6c57d6f5697, af9a152f4a8c4188ba8fb988fbfd7f5f; no voters:
I20260618 22:43:22.947623 24711 raft_consensus.cc:2781] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:22.947995 24711 raft_consensus.cc:686] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [term 1 LEADER]: Becoming Leader. State: Replica: af9a152f4a8c4188ba8fb988fbfd7f5f, State: Running, Role: LEADER
I20260618 22:43:22.948436 24711 consensus_queue.cc:237] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb703fabc14ddfae16d6c57d6f5697" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33599 } } peers { permanent_uuid: "af9a152f4a8c4188ba8fb988fbfd7f5f" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 41277 } } peers { permanent_uuid: "5691534cfe3f40509e88203fe41ab1a3" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 34453 } }
I20260618 22:43:22.953238 24230 catalog_manager.cc:5495] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f reported cstate change: term changed from 0 to 1, leader changed from <none> to af9a152f4a8c4188ba8fb988fbfd7f5f (127.5.213.193). New cstate: current_term: 1 leader_uuid: "af9a152f4a8c4188ba8fb988fbfd7f5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adb703fabc14ddfae16d6c57d6f5697" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af9a152f4a8c4188ba8fb988fbfd7f5f" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 41277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5691534cfe3f40509e88203fe41ab1a3" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 34453 } health_report { overall_health: UNKNOWN } } }
I20260618 22:43:23.015779 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:23.010405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.5.213.193:41277)
0618 22:43:23.010613 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:23.010619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:23.010696 (+ 77us) server_negotiation.cc:407] Connection header received
0618 22:43:23.010916 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:23.010920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:23.010998 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:23.011126 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:43:23.011521 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:23.012279 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:23.013615 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:23.014007 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:23.014111 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:23.014713 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:43:23.015199 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:43:23.015361 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:23.015510 (+ 149us) server_negotiation.cc:299] Negotiation successful
0618 22:43:23.015557 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260618 22:43:23.023675 24508 raft_consensus.cc:1261] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3 [term 1 FOLLOWER]: Refusing update from remote peer af9a152f4a8c4188ba8fb988fbfd7f5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:43:23.023676 24641 raft_consensus.cc:1261] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697 [term 1 FOLLOWER]: Refusing update from remote peer af9a152f4a8c4188ba8fb988fbfd7f5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:43:23.024600 24711 consensus_queue.cc:1035] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6adb703fabc14ddfae16d6c57d6f5697" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 33599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:43:23.024889 24715 consensus_queue.cc:1035] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5691534cfe3f40509e88203fe41ab1a3" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 34453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:43:23.036253 24719 mvcc.cc:205] Tried to move back new op lower bound from 7298345381975511040 to 7298345381680123904. Current Snapshot: MvccSnapshot[applied={T|T < 7298345381975511040}]
I20260618 22:43:23.040464 24718 mvcc.cc:205] Tried to move back new op lower bound from 7298345381975511040 to 7298345381680123904. Current Snapshot: MvccSnapshot[applied={T|T < 7298345381975511040 or (T in {7298345381975511040})}]
I20260618 22:43:23.049824 24227 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:43:23.050074 24227 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:43:23.053622 24227 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abdf8240f20142318b3fec3060cf2841: Sending DeleteTablet for 3 replicas of tablet 68c7023e105a4528bb9f569e9495ab85
I20260618 22:43:23.054738 24354 tablet_service.cc:1514] Processing DeleteTablet for tablet 68c7023e105a4528bb9f569e9495ab85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:43:23 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:52180
I20260618 22:43:23.054740 24621 tablet_service.cc:1514] Processing DeleteTablet for tablet 68c7023e105a4528bb9f569e9495ab85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:43:23 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:42996
I20260618 22:43:23.054934 24488 tablet_service.cc:1514] Processing DeleteTablet for tablet 68c7023e105a4528bb9f569e9495ab85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:43:23 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:49534
I20260618 22:43:23.059235 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 24291
I20260618 22:43:23.092684 24729 tablet_replica.cc:330] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697: stopping tablet replica
I20260618 22:43:23.093168 24729 raft_consensus.cc:2229] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:43:23.093731 24729 raft_consensus.cc:2258] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:43:23.096982 24728 tablet_replica.cc:330] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f: stopping tablet replica
I20260618 22:43:23.097433 24728 raft_consensus.cc:2229] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [term 1 LEADER]: Raft consensus shutting down.
Jun 18 22:43:23 dist-test-slave-lnj3 krb5kdc[24185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:43:23.103212 24730 tablet_replica.cc:330] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3: stopping tablet replica
I20260618 22:43:23.103963 24729 ts_tablet_manager.cc:1905] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:43:23.104005 24728 raft_consensus.cc:2258] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:43:23.104414 24730 raft_consensus.cc:2229] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:43:23.106655 24728 ts_tablet_manager.cc:1905] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:43:23.109376 24730 raft_consensus.cc:2258] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:43:23.111688 24729 ts_tablet_manager.cc:1918] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:43:23.111871 24729 log.cc:1196] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/68c7023e105a4528bb9f569e9495ab85
I20260618 22:43:23.112375 24729 ts_tablet_manager.cc:1939] T 68c7023e105a4528bb9f569e9495ab85 P 6adb703fabc14ddfae16d6c57d6f5697: Deleting consensus metadata
I20260618 22:43:23.113900 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:23.060655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52208 (local address 127.5.213.193:41277)
0618 22:43:23.067133 (+ 6478us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:23.067140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:23.085682 (+ 18542us) server_negotiation.cc:407] Connection header received
0618 22:43:23.086139 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:23.086145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:23.086229 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:23.086295 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:23.088165 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:23.089486 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:23.094234 (+ 4748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:23.095048 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:23.095153 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:23.104241 (+ 9088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:23.104259 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:23.104278 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:23.104321 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:23.108338 (+ 4017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:23.109794 (+ 1456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:23.109803 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:23.109807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:23.110011 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:23.110609 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:23.110615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:23.110618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:23.111095 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:23.111415 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:23.113154 (+ 1739us) server_negotiation.cc:299] Negotiation successful
0618 22:43:23.113432 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6299}
I20260618 22:43:23.114391 24214 catalog_manager.cc:4841] TS 6adb703fabc14ddfae16d6c57d6f5697 (127.5.213.195:33599): tablet 68c7023e105a4528bb9f569e9495ab85 (table test-table [id=c5b6ed9a549b474088303035d898c693]) successfully deleted
I20260618 22:43:23.114547 24728 ts_tablet_manager.cc:1918] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:43:23.114729 24728 log.cc:1196] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/68c7023e105a4528bb9f569e9495ab85
I20260618 22:43:23.115126 24730 ts_tablet_manager.cc:1905] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:43:23.115257 24728 ts_tablet_manager.cc:1939] T 68c7023e105a4528bb9f569e9495ab85 P af9a152f4a8c4188ba8fb988fbfd7f5f: Deleting consensus metadata
I20260618 22:43:23.115533 24414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:23.121183 24730 ts_tablet_manager.cc:1918] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:43:23.121351 24730 log.cc:1196] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/68c7023e105a4528bb9f569e9495ab85
I20260618 22:43:23.121737 24730 ts_tablet_manager.cc:1939] T 68c7023e105a4528bb9f569e9495ab85 P 5691534cfe3f40509e88203fe41ab1a3: Deleting consensus metadata
I20260618 22:43:23.123508 24216 catalog_manager.cc:4841] TS 5691534cfe3f40509e88203fe41ab1a3 (127.5.213.194:34453): tablet 68c7023e105a4528bb9f569e9495ab85 (table test-table [id=c5b6ed9a549b474088303035d898c693]) successfully deleted
I20260618 22:43:23.213814 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 24291
I20260618 22:43:23.214008 24217 catalog_manager.cc:4841] TS af9a152f4a8c4188ba8fb988fbfd7f5f (127.5.213.193:41277): tablet 68c7023e105a4528bb9f569e9495ab85 (table test-table [id=c5b6ed9a549b474088303035d898c693]) successfully deleted
I20260618 22:43:23.228710 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 24425
Jun 18 22:43:23 dist-test-slave-lnj3 krb5kdc[24185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.194@KRBTEST.COM
I20260618 22:43:23.243085 24702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:23.229496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49548 (local address 127.5.213.194:34453)
0618 22:43:23.229666 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:23.229670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:23.229686 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:43:23.229740 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:23.229744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:23.229805 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:23.229884 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:23.231163 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:23.231875 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:23.233170 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:23.233571 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:23.233677 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:23.237855 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:23.237871 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:23.237887 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:23.237929 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:23.240581 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:23.241264 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:23.241271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:23.241273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:23.241392 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:23.241666 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:23.241669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:23.241670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:23.242040 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:23.242257 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:23.242507 (+ 250us) server_negotiation.cc:299] Negotiation successful
0618 22:43:23.242805 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:43:23.243815 24548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:23.331274 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 24425
I20260618 22:43:23.344484 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 24558
Jun 18 22:43:23 dist-test-slave-lnj3 krb5kdc[24185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.195@KRBTEST.COM
I20260618 22:43:23.359577 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:23.345296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43006 (local address 127.5.213.195:33599)
0618 22:43:23.345469 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:23.345475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:23.345495 (+ 20us) server_negotiation.cc:407] Connection header received
0618 22:43:23.345674 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:23.345679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:23.345752 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:23.345872 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:23.347191 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:23.347895 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:23.349139 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:23.349533 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:23.349640 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:23.353838 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:23.353853 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:23.353868 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:23.353902 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:23.356719 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:23.357435 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:23.357444 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:23.357446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:23.357584 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:23.357946 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:23.357950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:23.357952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:23.358343 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:23.358902 (+ 559us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:23.359065 (+ 163us) server_negotiation.cc:299] Negotiation successful
0618 22:43:23.359296 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 22:43:23.360216 24681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:23.450084 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 24558
I20260618 22:43:23.463020 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 24198
I20260618 22:43:23.464061 24260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:23.572512 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 24198
2026-06-18T22:43:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4317 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-18T22:43:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:23Z Disabled control of system clock
I20260618 22:43:23.639441 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:42031
--webserver_interface=127.5.213.254
--webserver_port=0
--builtin_ntp_servers=127.5.213.212:36747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:42031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 22:43:23.952495 24739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:23.952782 24739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:23.952863 24739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:23.960086 24739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:43:23.960199 24739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:23.960237 24739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:23.960292 24739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:43:23.960338 24739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:43:23.969419 24739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:36747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:42031
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:42031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:23.970674 24739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:23.972370 24739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:23.983402 24745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:23.983626 24744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:23.983381 24747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:23.985433 24739 server_base.cc:1034] running on GCE node
I20260618 22:43:23.986339 24739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:23.987807 24739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:23.989046 24739 hybrid_clock.cc:648] HybridClock initialized: now 1781822603988989 us; error 38 us; skew 500 ppm
I20260618 22:43:23.989472 24739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:43:23.991719 24739 webserver.cc:460] Webserver started at http://127.5.213.254:32919/ using document root <none> and password file <none>
I20260618 22:43:23.992380 24739 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:23.992468 24739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:23.992776 24739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:23.995705 24739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "a641b54a809048ccafc7e93bfad28fda"
format_stamp: "Formatted at 2026-06-18 22:43:23 on dist-test-slave-lnj3"
server_key: "50f531f878cb98c6692272f578bb619a"
server_key_iv: "82fdd019ee12ec1db04fe5032f2ca1b3"
server_key_version: "encryptionkey@0"
I20260618 22:43:23.996527 24739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "a641b54a809048ccafc7e93bfad28fda"
format_stamp: "Formatted at 2026-06-18 22:43:23 on dist-test-slave-lnj3"
server_key: "50f531f878cb98c6692272f578bb619a"
server_key_iv: "82fdd019ee12ec1db04fe5032f2ca1b3"
server_key_version: "encryptionkey@0"
I20260618 22:43:24.001958 24739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260618 22:43:24.005556 24753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:24.006902 24739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:43:24.007083 24739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "a641b54a809048ccafc7e93bfad28fda"
format_stamp: "Formatted at 2026-06-18 22:43:23 on dist-test-slave-lnj3"
server_key: "50f531f878cb98c6692272f578bb619a"
server_key_iv: "82fdd019ee12ec1db04fe5032f2ca1b3"
server_key_version: "encryptionkey@0"
I20260618 22:43:24.007282 24739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:24.020744 24739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:24.021490 24739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:24.021756 24739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:24.038450 24739 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:42031
I20260618 22:43:24.038470 24804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:42031 every 8 connection(s)
I20260618 22:43:24.039791 24739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:43:24.040496 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 24739
I20260618 22:43:24.040660 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:43:24.041210 5975 external_mini_cluster.cc:1428] Setting key 7adf1bd252e1b2ec430858df52914bb0
I20260618 22:43:24.044067 24805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:24.054585 24808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:24.043143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40112 (local address 127.5.213.254:42031)
0618 22:43:24.044071 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:24.044082 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:24.044134 (+ 52us) server_negotiation.cc:407] Connection header received
0618 22:43:24.045746 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:24.045766 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:24.046233 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:24.046558 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:24.047157 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:24.048425 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:24.050256 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:24.050997 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:24.051161 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:24.052363 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:24.052370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:24.052387 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:24.052927 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:24.053010 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:24.053032 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:24.053132 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:24.053337 (+ 205us) server_negotiation.cc:299] Negotiation successful
0618 22:43:24.053651 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":320,"threads_started":1}
I20260618 22:43:24.055714 24805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda: Bootstrap starting.
I20260618 22:43:24.058548 24805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:24.059511 24805 log.cc:824] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:24.061869 24805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda: No bootstrap required, opened a new log
I20260618 22:43:24.068519 24805 raft_consensus.cc:348] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a641b54a809048ccafc7e93bfad28fda" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42031 } }
I20260618 22:43:24.068841 24805 raft_consensus.cc:374] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:24.068924 24805 raft_consensus.cc:729] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a641b54a809048ccafc7e93bfad28fda, State: Initialized, Role: FOLLOWER
I20260618 22:43:24.069418 24805 consensus_queue.cc:260] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a641b54a809048ccafc7e93bfad28fda" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42031 } }
I20260618 22:43:24.069625 24805 raft_consensus.cc:388] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:43:24.069746 24805 raft_consensus.cc:482] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:43:24.069900 24805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:24.072000 24805 raft_consensus.cc:504] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a641b54a809048ccafc7e93bfad28fda" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42031 } }
I20260618 22:43:24.072448 24805 leader_election.cc:302] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a641b54a809048ccafc7e93bfad28fda; no voters:
I20260618 22:43:24.072975 24805 leader_election.cc:288] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:43:24.073088 24810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:24.073719 24810 raft_consensus.cc:686] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda [term 1 LEADER]: Becoming Leader. State: Replica: a641b54a809048ccafc7e93bfad28fda, State: Running, Role: LEADER
I20260618 22:43:24.074258 24810 consensus_queue.cc:237] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a641b54a809048ccafc7e93bfad28fda" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42031 } }
I20260618 22:43:24.074806 24805 sys_catalog.cc:564] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:43:24.076759 24811 sys_catalog.cc:455] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a641b54a809048ccafc7e93bfad28fda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a641b54a809048ccafc7e93bfad28fda" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42031 } } }
I20260618 22:43:24.076970 24812 sys_catalog.cc:455] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda [sys.catalog]: SysCatalogTable state changed. Reason: New leader a641b54a809048ccafc7e93bfad28fda. Latest consensus state: current_term: 1 leader_uuid: "a641b54a809048ccafc7e93bfad28fda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a641b54a809048ccafc7e93bfad28fda" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42031 } } }
I20260618 22:43:24.077153 24811 sys_catalog.cc:458] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:24.077215 24812 sys_catalog.cc:458] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:24.079115 24818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:43:24.084240 24818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:43:24.092730 24818 catalog_manager.cc:1347] Generated new cluster ID: 7ab79bbf39374c9e83bde4fd9acfe8f3
I20260618 22:43:24.092823 24818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:43:24.103281 24818 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:43:24.104530 24818 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:43:24.118811 24818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda: Generated new TSK 0
I20260618 22:43:24.119987 24818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:43:24.136647 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:42031
--builtin_ntp_servers=127.5.213.212:36747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 22:43:24.472358 24829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:24.472752 24829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:24.472894 24829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:24.480741 24829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:24.480883 24829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:24.481045 24829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:43:24.490900 24829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:36747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:42031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:24.492291 24829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:24.494230 24829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:24.508260 24834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:24.508213 24835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:24.508175 24837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:24.509756 24829 server_base.cc:1034] running on GCE node
I20260618 22:43:24.510479 24829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:24.511564 24829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:24.512773 24829 hybrid_clock.cc:648] HybridClock initialized: now 1781822604512717 us; error 49 us; skew 500 ppm
I20260618 22:43:24.513109 24829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:43:24.515654 24829 webserver.cc:460] Webserver started at http://127.5.213.193:46815/ using document root <none> and password file <none>
I20260618 22:43:24.516435 24829 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:24.516547 24829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:24.516880 24829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:24.520081 24829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "1833fc7638584bb9aea18d59621bd0e3"
format_stamp: "Formatted at 2026-06-18 22:43:24 on dist-test-slave-lnj3"
server_key: "1bec8d3f23759277da65193d15c35575"
server_key_iv: "b036641c4304b9c97919f489f8226c5c"
server_key_version: "encryptionkey@0"
I20260618 22:43:24.520941 24829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "1833fc7638584bb9aea18d59621bd0e3"
format_stamp: "Formatted at 2026-06-18 22:43:24 on dist-test-slave-lnj3"
server_key: "1bec8d3f23759277da65193d15c35575"
server_key_iv: "b036641c4304b9c97919f489f8226c5c"
server_key_version: "encryptionkey@0"
I20260618 22:43:24.526757 24829 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260618 22:43:24.530897 24843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:24.532491 24829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:43:24.532670 24829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "1833fc7638584bb9aea18d59621bd0e3"
format_stamp: "Formatted at 2026-06-18 22:43:24 on dist-test-slave-lnj3"
server_key: "1bec8d3f23759277da65193d15c35575"
server_key_iv: "b036641c4304b9c97919f489f8226c5c"
server_key_version: "encryptionkey@0"
I20260618 22:43:24.532919 24829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:24.551254 24829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:24.552166 24829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:24.552467 24829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:24.553875 24829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:43:24.555819 24829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:43:24.555922 24829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:24.556006 24829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:43:24.556058 24829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:24.588372 24829 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:36179
I20260618 22:43:24.588419 24955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:36179 every 8 connection(s)
I20260618 22:43:24.589751 24829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:43:24.593663 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 24829
I20260618 22:43:24.593834 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:43:24.594360 5975 external_mini_cluster.cc:1428] Setting key 31c6a715095fb85df04f33173fe97f5f
I20260618 22:43:24.597255 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:42031
--builtin_ntp_servers=127.5.213.212:36747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 22:43:24.603071 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:24.592424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:49391 (local address 127.5.213.254:42031)
0618 22:43:24.592926 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:24.592932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:24.594239 (+ 1307us) server_negotiation.cc:407] Connection header received
0618 22:43:24.596557 (+ 2318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:24.596562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:24.596656 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:24.596751 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:24.597978 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:24.598746 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:24.600308 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:24.600709 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:24.600818 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:24.602177 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:24.602183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:24.602186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:24.602214 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:24.602260 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:24.602272 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:24.602347 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:24.602562 (+ 215us) server_negotiation.cc:299] Negotiation successful
0618 22:43:24.602728 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":225,"threads_started":1}
I20260618 22:43:24.604130 24959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:24.593039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:42031 (local address 127.5.213.193:49391)
0618 22:43:24.594071 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:24.594117 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:24.596233 (+ 2116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:24.597032 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:24.597040 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:24.597144 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:43:24.597814 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:24.597833 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:24.598884 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:24.598888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:24.600166 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:24.600178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:24.600928 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:24.600935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:24.601172 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:24.601844 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:43:24.601847 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:43:24.601895 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:43:24.601983 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:43:24.602362 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:43:24.602374 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:43:24.602382 (+ 8us) client_negotiation.cc:769] Sending connection context
0618 22:43:24.602540 (+ 158us) client_negotiation.cc:240] Negotiation successful
0618 22:43:24.603012 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":289,"threads_started":1}
I20260618 22:43:24.605767 24956 heartbeater.cc:347] Connected to a master server at 127.5.213.254:42031
I20260618 22:43:24.606137 24956 heartbeater.cc:464] Registering TS with master...
I20260618 22:43:24.607170 24956 heartbeater.cc:511] Master 127.5.213.254:42031 requested a full tablet report, sending...
I20260618 22:43:24.609704 24770 ts_manager.cc:194] Registered new tserver with Master: 1833fc7638584bb9aea18d59621bd0e3 (127.5.213.193:36179)
I20260618 22:43:24.611950 24770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.213.193:49391
W20260618 22:43:24.921334 24960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:24.921654 24960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:24.921762 24960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:24.929315 24960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:24.929446 24960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:24.929621 24960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:43:24.938962 24960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:36747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:42031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:24.940274 24960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:24.942059 24960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:24.954901 24966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:24.954960 24968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:24.954890 24965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:24.955543 24960 server_base.cc:1034] running on GCE node
I20260618 22:43:24.956097 24960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:24.957011 24960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:24.958245 24960 hybrid_clock.cc:648] HybridClock initialized: now 1781822604958183 us; error 45 us; skew 500 ppm
I20260618 22:43:24.958561 24960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:43:24.960839 24960 webserver.cc:460] Webserver started at http://127.5.213.194:43667/ using document root <none> and password file <none>
I20260618 22:43:24.961450 24960 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:24.961544 24960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:24.961855 24960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:24.964646 24960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "875aeafdea04483e960993e40e2515c8"
format_stamp: "Formatted at 2026-06-18 22:43:24 on dist-test-slave-lnj3"
server_key: "27cd2a85d94feb4209fad2cf1a5cfa7c"
server_key_iv: "1f9b5cb57b14c76186cab869c72f336c"
server_key_version: "encryptionkey@0"
I20260618 22:43:24.965652 24960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "875aeafdea04483e960993e40e2515c8"
format_stamp: "Formatted at 2026-06-18 22:43:24 on dist-test-slave-lnj3"
server_key: "27cd2a85d94feb4209fad2cf1a5cfa7c"
server_key_iv: "1f9b5cb57b14c76186cab869c72f336c"
server_key_version: "encryptionkey@0"
I20260618 22:43:24.970747 24960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260618 22:43:24.974139 24974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:24.975478 24960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:43:24.975607 24960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "875aeafdea04483e960993e40e2515c8"
format_stamp: "Formatted at 2026-06-18 22:43:24 on dist-test-slave-lnj3"
server_key: "27cd2a85d94feb4209fad2cf1a5cfa7c"
server_key_iv: "1f9b5cb57b14c76186cab869c72f336c"
server_key_version: "encryptionkey@0"
I20260618 22:43:24.975870 24960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:24.996454 24960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:24.997274 24960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:24.997560 24960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:24.998510 24960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:43:25.000254 24960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:43:25.000370 24960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:25.000485 24960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:43:25.000563 24960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:25.030951 24960 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:42069
I20260618 22:43:25.031013 25086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:42069 every 8 connection(s)
I20260618 22:43:25.032267 24960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:43:25.033097 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 24960
I20260618 22:43:25.033262 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:43:25.033756 5975 external_mini_cluster.cc:1428] Setting key 0de700aff365c16823d0f8e53076d056
I20260618 22:43:25.036720 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:42031
--builtin_ntp_servers=127.5.213.212:36747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 22:43:25.045133 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:25.034816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:34029 (local address 127.5.213.254:42031)
0618 22:43:25.035004 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:25.035009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:25.036526 (+ 1517us) server_negotiation.cc:407] Connection header received
0618 22:43:25.038810 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:25.038816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:25.038900 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:25.038986 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:25.040069 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.040830 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.042364 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.042897 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.043012 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:25.044331 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:25.044337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:25.044340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:25.044358 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:25.044393 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:25.044408 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:25.044482 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:25.044701 (+ 219us) server_negotiation.cc:299] Negotiation successful
0618 22:43:25.044802 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 22:43:25.046185 25089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:25.035390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:42031 (local address 127.5.213.194:34029)
0618 22:43:25.036310 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:25.036358 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:25.038478 (+ 2120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:25.039238 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:25.039246 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:25.039342 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:43:25.039901 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:25.039917 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.040982 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.040986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:25.042223 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:25.042235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.043154 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.043162 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:25.043381 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:25.044007 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:43:25.044010 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:43:25.044049 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:43:25.044139 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:43:25.044496 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:43:25.044506 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:43:25.044514 (+ 8us) client_negotiation.cc:769] Sending connection context
0618 22:43:25.044670 (+ 156us) client_negotiation.cc:240] Negotiation successful
0618 22:43:25.045085 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":292,"threads_started":1}
I20260618 22:43:25.047578 25087 heartbeater.cc:347] Connected to a master server at 127.5.213.254:42031
I20260618 22:43:25.047883 25087 heartbeater.cc:464] Registering TS with master...
I20260618 22:43:25.048697 25087 heartbeater.cc:511] Master 127.5.213.254:42031 requested a full tablet report, sending...
I20260618 22:43:25.050199 24770 ts_manager.cc:194] Registered new tserver with Master: 875aeafdea04483e960993e40e2515c8 (127.5.213.194:42069)
I20260618 22:43:25.051236 24770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.213.194:34029
W20260618 22:43:25.352767 25090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:25.353040 25090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:25.353149 25090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:25.360394 25090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:25.360503 25090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:25.360649 25090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:43:25.369699 25090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:36747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:42031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:25.370949 25090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:25.372651 25090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:25.385046 25096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:25.385071 25095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:25.385156 25098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:25.385655 25090 server_base.cc:1034] running on GCE node
I20260618 22:43:25.386276 25090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:25.387204 25090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:25.388427 25090 hybrid_clock.cc:648] HybridClock initialized: now 1781822605388365 us; error 44 us; skew 500 ppm
I20260618 22:43:25.388737 25090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:43:25.391044 25090 webserver.cc:460] Webserver started at http://127.5.213.195:42933/ using document root <none> and password file <none>
I20260618 22:43:25.391675 25090 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:25.391757 25090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:25.392011 25090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:25.394774 25090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "92b712c0575245ffa417da675a79ed8a"
format_stamp: "Formatted at 2026-06-18 22:43:25 on dist-test-slave-lnj3"
server_key: "ecb062d5c0f9e6ecf0c87e78ae3a3744"
server_key_iv: "f69fee16ac1c971f7f61cf541a3f9f68"
server_key_version: "encryptionkey@0"
I20260618 22:43:25.395471 25090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "92b712c0575245ffa417da675a79ed8a"
format_stamp: "Formatted at 2026-06-18 22:43:25 on dist-test-slave-lnj3"
server_key: "ecb062d5c0f9e6ecf0c87e78ae3a3744"
server_key_iv: "f69fee16ac1c971f7f61cf541a3f9f68"
server_key_version: "encryptionkey@0"
I20260618 22:43:25.400249 25090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260618 22:43:25.403512 25104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:25.404801 25090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:43:25.404918 25090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "92b712c0575245ffa417da675a79ed8a"
format_stamp: "Formatted at 2026-06-18 22:43:25 on dist-test-slave-lnj3"
server_key: "ecb062d5c0f9e6ecf0c87e78ae3a3744"
server_key_iv: "f69fee16ac1c971f7f61cf541a3f9f68"
server_key_version: "encryptionkey@0"
I20260618 22:43:25.405081 25090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:25.427362 25090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:25.428079 25090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:25.428325 25090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:25.429158 25090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:43:25.430459 25090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:43:25.430539 25090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:25.430610 25090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:43:25.430683 25090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260618 22:43:25.460157 25090 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:34543
I20260618 22:43:25.460211 25216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:34543 every 8 connection(s)
I20260618 22:43:25.461503 25090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:43:25.462913 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 25090
I20260618 22:43:25.463050 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:43:25.463529 5975 external_mini_cluster.cc:1428] Setting key c69a48ffead3ccc6dae2545284101d6e
I20260618 22:43:25.474277 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:25.464037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:41837 (local address 127.5.213.254:42031)
0618 22:43:25.464246 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:25.464252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:25.466404 (+ 2152us) server_negotiation.cc:407] Connection header received
0618 22:43:25.468173 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:25.468177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:25.468245 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:25.468324 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:25.469306 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.470028 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.471775 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.472169 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.472271 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:25.473592 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:25.473597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:25.473600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:25.473617 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:25.473648 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:25.473695 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:25.473759 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:25.473938 (+ 179us) server_negotiation.cc:299] Negotiation successful
0618 22:43:25.474016 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260618 22:43:25.475353 25219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:25.464614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:42031 (local address 127.5.213.195:41837)
0618 22:43:25.466278 (+ 1664us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:25.466318 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:25.467929 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:25.468504 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:25.468512 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:25.468607 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:43:25.469169 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:25.469186 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.470232 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.470236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:25.471612 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:25.471626 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.472356 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.472361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:25.472564 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:25.473260 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:43:25.473263 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:43:25.473301 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:43:25.473398 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:43:25.473771 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:43:25.473780 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:43:25.473785 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:43:25.473925 (+ 140us) client_negotiation.cc:240] Negotiation successful
0618 22:43:25.474237 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1186,"thread_start_us":414,"threads_started":1}
I20260618 22:43:25.476785 25217 heartbeater.cc:347] Connected to a master server at 127.5.213.254:42031
I20260618 22:43:25.477105 25217 heartbeater.cc:464] Registering TS with master...
I20260618 22:43:25.478004 25217 heartbeater.cc:511] Master 127.5.213.254:42031 requested a full tablet report, sending...
I20260618 22:43:25.479490 24770 ts_manager.cc:194] Registered new tserver with Master: 92b712c0575245ffa417da675a79ed8a (127.5.213.195:34543)
I20260618 22:43:25.480350 24770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.213.195:41837
I20260618 22:43:25.490689 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:43:25.500846 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:25.494458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.5.213.254:42031)
0618 22:43:25.494744 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:25.494754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:25.495140 (+ 386us) server_negotiation.cc:407] Connection header received
0618 22:43:25.495266 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:25.495270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:25.495334 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:25.495410 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:25.495972 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.497007 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.498328 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.498806 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.498911 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:25.500132 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:25.500136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:25.500139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:25.500158 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:25.500201 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:25.500213 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:25.500327 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:25.500489 (+ 162us) server_negotiation.cc:299] Negotiation successful
0618 22:43:25.500555 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260618 22:43:25.504835 24770 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:43:25.512054 24770 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:43:25.535660 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:25.525311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:36179 (local address 127.0.0.1:59292)
0618 22:43:25.527205 (+ 1894us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:25.527224 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:25.527338 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:25.528654 (+ 1316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:25.528659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:25.528692 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:25.530016 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:25.530028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.531485 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.531490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:25.533260 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:25.533274 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.534681 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.534688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:25.534871 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:25.534921 (+ 50us) client_negotiation.cc:769] Sending connection context
0618 22:43:25.535051 (+ 130us) client_negotiation.cc:240] Negotiation successful
0618 22:43:25.535250 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1771,"mutex_wait_us":869,"thread_start_us":1607,"threads_started":1}
I20260618 22:43:25.535820 25229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:25.525129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:42069 (local address 127.0.0.1:57676)
0618 22:43:25.526970 (+ 1841us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:25.526988 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:25.527111 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:25.528236 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:25.528242 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:25.528281 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:25.528747 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:25.528765 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.530453 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.530458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:25.533329 (+ 2871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:25.533340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.535051 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.535058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:25.535250 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:25.535280 (+ 30us) client_negotiation.cc:769] Sending connection context
0618 22:43:25.535364 (+ 84us) client_negotiation.cc:240] Negotiation successful
0618 22:43:25.535480 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1692,"mutex_wait_us":133,"spinlock_wait_cycles":1280,"thread_start_us":388,"threads_started":1}
I20260618 22:43:25.535831 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:25.525734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57676 (local address 127.5.213.194:42069)
0618 22:43:25.527673 (+ 1939us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:25.527683 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:25.527712 (+ 29us) server_negotiation.cc:407] Connection header received
0618 22:43:25.527816 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:25.527823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:25.528044 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:25.528207 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:25.529048 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.530341 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.533513 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.534865 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.534996 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:25.535256 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:25.535400 (+ 144us) server_negotiation.cc:299] Negotiation successful
0618 22:43:25.535530 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1760,"thread_start_us":696,"threads_started":1}
I20260618 22:43:25.536883 25231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:25.524484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:34543 (local address 127.0.0.1:55590)
0618 22:43:25.525955 (+ 1471us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:25.525997 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:25.526169 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:25.528274 (+ 2105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:25.528278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:25.528302 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:25.528895 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:25.528909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.530620 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.530646 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:25.533182 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:25.533198 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.536053 (+ 2855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.536060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:25.536233 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:25.536265 (+ 32us) client_negotiation.cc:769] Sending connection context
0618 22:43:25.536366 (+ 101us) client_negotiation.cc:240] Negotiation successful
0618 22:43:25.536503 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1167,"mutex_wait_us":99,"thread_start_us":2141,"threads_started":1}
I20260618 22:43:25.536981 25233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:25.525513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59292 (local address 127.5.213.193:36179)
0618 22:43:25.527933 (+ 2420us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:25.527940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:25.527964 (+ 24us) server_negotiation.cc:407] Connection header received
0618 22:43:25.528053 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:25.528059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:25.528261 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:25.528435 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:25.530193 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.531346 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.533494 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.534479 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.534598 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:25.534875 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:25.536118 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0618 22:43:25.536302 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2282,"thread_start_us":2160,"threads_started":1}
I20260618 22:43:25.538035 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:25.524803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55590 (local address 127.5.213.195:34543)
0618 22:43:25.527665 (+ 2862us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:25.527680 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:25.527715 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:43:25.527814 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:25.527822 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:25.528069 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:25.528960 (+ 891us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:25.529230 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.530478 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.533423 (+ 2945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.534865 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.534996 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:25.535262 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:25.536414 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0618 22:43:25.536581 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2693,"thread_start_us":2549,"threads_started":1}
I20260618 22:43:25.540108 25022 tablet_service.cc:1467] Processing CreateTablet for tablet f43e035b0c2744cb88651b34cef1cfe0 (DEFAULT_TABLE table=test-table [id=d31f01dbc102467ea6ee2e940afb1133]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:43:25.540761 24891 tablet_service.cc:1467] Processing CreateTablet for tablet f43e035b0c2744cb88651b34cef1cfe0 (DEFAULT_TABLE table=test-table [id=d31f01dbc102467ea6ee2e940afb1133]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:43:25.541024 25152 tablet_service.cc:1467] Processing CreateTablet for tablet f43e035b0c2744cb88651b34cef1cfe0 (DEFAULT_TABLE table=test-table [id=d31f01dbc102467ea6ee2e940afb1133]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:43:25.541617 25022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f43e035b0c2744cb88651b34cef1cfe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:25.542124 24891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f43e035b0c2744cb88651b34cef1cfe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:25.542289 25152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f43e035b0c2744cb88651b34cef1cfe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:25.555866 25235 tablet_bootstrap.cc:492] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a: Bootstrap starting.
I20260618 22:43:25.558233 25236 tablet_bootstrap.cc:492] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3: Bootstrap starting.
I20260618 22:43:25.559159 25237 tablet_bootstrap.cc:492] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8: Bootstrap starting.
I20260618 22:43:25.561805 25235 tablet_bootstrap.cc:654] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:25.563050 25237 tablet_bootstrap.cc:654] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:25.563241 25235 log.cc:824] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:25.564333 25237 log.cc:824] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:25.565893 25236 tablet_bootstrap.cc:654] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:25.566676 25235 tablet_bootstrap.cc:492] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a: No bootstrap required, opened a new log
I20260618 22:43:25.567008 25235 ts_tablet_manager.cc:1397] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260618 22:43:25.566944 25236 log.cc:824] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:25.567514 25237 tablet_bootstrap.cc:492] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8: No bootstrap required, opened a new log
I20260618 22:43:25.567847 25237 ts_tablet_manager.cc:1397] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260618 22:43:25.570082 25236 tablet_bootstrap.cc:492] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3: No bootstrap required, opened a new log
I20260618 22:43:25.570320 25236 ts_tablet_manager.cc:1397] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20260618 22:43:25.577256 25236 raft_consensus.cc:348] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b712c0575245ffa417da675a79ed8a" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34543 } } peers { permanent_uuid: "1833fc7638584bb9aea18d59621bd0e3" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36179 } } peers { permanent_uuid: "875aeafdea04483e960993e40e2515c8" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 42069 } }
I20260618 22:43:25.577874 25236 raft_consensus.cc:374] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:25.578052 25236 raft_consensus.cc:729] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1833fc7638584bb9aea18d59621bd0e3, State: Initialized, Role: FOLLOWER
I20260618 22:43:25.578171 25235 raft_consensus.cc:348] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b712c0575245ffa417da675a79ed8a" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34543 } } peers { permanent_uuid: "1833fc7638584bb9aea18d59621bd0e3" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36179 } } peers { permanent_uuid: "875aeafdea04483e960993e40e2515c8" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 42069 } }
I20260618 22:43:25.578761 25235 raft_consensus.cc:374] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:25.578434 25237 raft_consensus.cc:348] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b712c0575245ffa417da675a79ed8a" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34543 } } peers { permanent_uuid: "1833fc7638584bb9aea18d59621bd0e3" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36179 } } peers { permanent_uuid: "875aeafdea04483e960993e40e2515c8" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 42069 } }
I20260618 22:43:25.578912 25237 raft_consensus.cc:374] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:25.578955 25235 raft_consensus.cc:729] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b712c0575245ffa417da675a79ed8a, State: Initialized, Role: FOLLOWER
I20260618 22:43:25.579085 25236 consensus_queue.cc:260] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b712c0575245ffa417da675a79ed8a" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34543 } } peers { permanent_uuid: "1833fc7638584bb9aea18d59621bd0e3" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36179 } } peers { permanent_uuid: "875aeafdea04483e960993e40e2515c8" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 42069 } }
I20260618 22:43:25.579562 25237 raft_consensus.cc:729] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875aeafdea04483e960993e40e2515c8, State: Initialized, Role: FOLLOWER
I20260618 22:43:25.579770 25235 consensus_queue.cc:260] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b712c0575245ffa417da675a79ed8a" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34543 } } peers { permanent_uuid: "1833fc7638584bb9aea18d59621bd0e3" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36179 } } peers { permanent_uuid: "875aeafdea04483e960993e40e2515c8" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 42069 } }
I20260618 22:43:25.580379 25237 consensus_queue.cc:260] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b712c0575245ffa417da675a79ed8a" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34543 } } peers { permanent_uuid: "1833fc7638584bb9aea18d59621bd0e3" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36179 } } peers { permanent_uuid: "875aeafdea04483e960993e40e2515c8" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 42069 } }
I20260618 22:43:25.580694 24956 heartbeater.cc:503] Master 127.5.213.254:42031 was elected leader, sending a full tablet report...
I20260618 22:43:25.581400 25236 ts_tablet_manager.cc:1428] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260618 22:43:25.582969 25087 heartbeater.cc:503] Master 127.5.213.254:42031 was elected leader, sending a full tablet report...
I20260618 22:43:25.583611 25237 ts_tablet_manager.cc:1428] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260618 22:43:25.584113 25217 heartbeater.cc:503] Master 127.5.213.254:42031 was elected leader, sending a full tablet report...
I20260618 22:43:25.584585 25235 ts_tablet_manager.cc:1428] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
W20260618 22:43:25.591905 24957 tablet.cc:2369] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:43:25.713066 25218 tablet.cc:2369] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:43:25.784228 25088 tablet.cc:2369] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:43:25.867019 25241 raft_consensus.cc:482] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:43:25.867452 25241 raft_consensus.cc:504] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b712c0575245ffa417da675a79ed8a" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34543 } } peers { permanent_uuid: "1833fc7638584bb9aea18d59621bd0e3" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36179 } } peers { permanent_uuid: "875aeafdea04483e960993e40e2515c8" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 42069 } }
I20260618 22:43:25.869756 25241 leader_election.cc:288] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92b712c0575245ffa417da675a79ed8a (127.5.213.195:34543), 875aeafdea04483e960993e40e2515c8 (127.5.213.194:42069)
I20260618 22:43:25.878517 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:25.870222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:50301 (local address 127.5.213.194:42069)
0618 22:43:25.870932 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:25.870938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:25.871926 (+ 988us) server_negotiation.cc:407] Connection header received
0618 22:43:25.872068 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:25.872072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:25.872157 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:25.872237 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:25.872962 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.873981 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.876296 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.877466 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.877637 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:25.877713 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:25.878073 (+ 360us) server_negotiation.cc:299] Negotiation successful
0618 22:43:25.878227 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547}
I20260618 22:43:25.878767 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:25.870500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:38403 (local address 127.5.213.195:34543)
0618 22:43:25.870732 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:25.870739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:25.871593 (+ 854us) server_negotiation.cc:407] Connection header received
0618 22:43:25.871912 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:25.871918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:25.872003 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:25.872114 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:25.872744 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.873830 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.876624 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.877850 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.877997 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:25.878064 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:25.878351 (+ 287us) server_negotiation.cc:299] Negotiation successful
0618 22:43:25.878477 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260618 22:43:25.878821 25244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:25.870240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:34543 (local address 127.5.213.193:38403)
0618 22:43:25.871473 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:25.871492 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:25.871728 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:25.872147 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:25.872151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:25.872174 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:25.872584 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:25.872596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.874006 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.874011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:25.876410 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:25.876426 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.878026 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.878032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:25.878193 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:25.878216 (+ 23us) client_negotiation.cc:769] Sending connection context
0618 22:43:25.878294 (+ 78us) client_negotiation.cc:240] Negotiation successful
0618 22:43:25.878428 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"mutex_wait_us":23,"thread_start_us":899,"threads_started":1}
I20260618 22:43:25.879493 25245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:25.870448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:42069 (local address 127.5.213.193:50301)
0618 22:43:25.871826 (+ 1378us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:25.871841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:25.871950 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:25.872311 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:25.872316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:25.872345 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:25.872775 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:25.872786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.874129 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.874170 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:25.876092 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:25.876110 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.877673 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.877680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:25.877863 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:25.877902 (+ 39us) client_negotiation.cc:769] Sending connection context
0618 22:43:25.878981 (+ 1079us) client_negotiation.cc:240] Negotiation successful
0618 22:43:25.879133 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1266,"mutex_wait_us":209,"thread_start_us":1058,"threads_started":1}
I20260618 22:43:25.879930 25172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f43e035b0c2744cb88651b34cef1cfe0" candidate_uuid: "1833fc7638584bb9aea18d59621bd0e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b712c0575245ffa417da675a79ed8a" is_pre_election: true
I20260618 22:43:25.880486 25172 raft_consensus.cc:2454] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1833fc7638584bb9aea18d59621bd0e3 in term 0.
I20260618 22:43:25.880584 25042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f43e035b0c2744cb88651b34cef1cfe0" candidate_uuid: "1833fc7638584bb9aea18d59621bd0e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875aeafdea04483e960993e40e2515c8" is_pre_election: true
I20260618 22:43:25.881140 25042 raft_consensus.cc:2454] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1833fc7638584bb9aea18d59621bd0e3 in term 0.
I20260618 22:43:25.881196 24845 leader_election.cc:302] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1833fc7638584bb9aea18d59621bd0e3, 92b712c0575245ffa417da675a79ed8a; no voters:
I20260618 22:43:25.881585 25241 raft_consensus.cc:2781] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:43:25.881757 25241 raft_consensus.cc:482] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:43:25.881871 25241 raft_consensus.cc:3037] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:25.884105 25241 raft_consensus.cc:504] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b712c0575245ffa417da675a79ed8a" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34543 } } peers { permanent_uuid: "1833fc7638584bb9aea18d59621bd0e3" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36179 } } peers { permanent_uuid: "875aeafdea04483e960993e40e2515c8" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 42069 } }
I20260618 22:43:25.884716 25241 leader_election.cc:288] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [CANDIDATE]: Term 1 election: Requested vote from peers 92b712c0575245ffa417da675a79ed8a (127.5.213.195:34543), 875aeafdea04483e960993e40e2515c8 (127.5.213.194:42069)
I20260618 22:43:25.885151 25172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f43e035b0c2744cb88651b34cef1cfe0" candidate_uuid: "1833fc7638584bb9aea18d59621bd0e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b712c0575245ffa417da675a79ed8a"
I20260618 22:43:25.885241 25042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f43e035b0c2744cb88651b34cef1cfe0" candidate_uuid: "1833fc7638584bb9aea18d59621bd0e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875aeafdea04483e960993e40e2515c8"
I20260618 22:43:25.885366 25172 raft_consensus.cc:3037] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:25.885443 25042 raft_consensus.cc:3037] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:25.888494 25172 raft_consensus.cc:2454] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1833fc7638584bb9aea18d59621bd0e3 in term 1.
I20260618 22:43:25.888509 25042 raft_consensus.cc:2454] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1833fc7638584bb9aea18d59621bd0e3 in term 1.
I20260618 22:43:25.888902 24847 leader_election.cc:302] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1833fc7638584bb9aea18d59621bd0e3, 875aeafdea04483e960993e40e2515c8; no voters:
I20260618 22:43:25.889230 25241 raft_consensus.cc:2781] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:25.889837 25241 raft_consensus.cc:686] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [term 1 LEADER]: Becoming Leader. State: Replica: 1833fc7638584bb9aea18d59621bd0e3, State: Running, Role: LEADER
I20260618 22:43:25.890441 25241 consensus_queue.cc:237] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b712c0575245ffa417da675a79ed8a" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34543 } } peers { permanent_uuid: "1833fc7638584bb9aea18d59621bd0e3" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36179 } } peers { permanent_uuid: "875aeafdea04483e960993e40e2515c8" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 42069 } }
I20260618 22:43:25.896023 24769 catalog_manager.cc:5495] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1833fc7638584bb9aea18d59621bd0e3 (127.5.213.193). New cstate: current_term: 1 leader_uuid: "1833fc7638584bb9aea18d59621bd0e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b712c0575245ffa417da675a79ed8a" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1833fc7638584bb9aea18d59621bd0e3" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "875aeafdea04483e960993e40e2515c8" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 42069 } health_report { overall_health: UNKNOWN } } }
I20260618 22:43:25.998545 25233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:25.992030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59298 (local address 127.5.213.193:36179)
0618 22:43:25.992210 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:25.992215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:25.992468 (+ 253us) server_negotiation.cc:407] Connection header received
0618 22:43:25.992671 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:25.992709 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:25.992803 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:25.993038 (+ 235us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:43:25.993481 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.994231 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.996268 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:25.996716 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:25.996812 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:25.997369 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:43:25.997924 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:43:25.998050 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:25.998200 (+ 150us) server_negotiation.cc:299] Negotiation successful
0618 22:43:25.998239 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 22:43:26.007237 25042 raft_consensus.cc:1261] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8 [term 1 FOLLOWER]: Refusing update from remote peer 1833fc7638584bb9aea18d59621bd0e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:43:26.007232 25172 raft_consensus.cc:1261] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a [term 1 FOLLOWER]: Refusing update from remote peer 1833fc7638584bb9aea18d59621bd0e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:43:26.008352 25241 consensus_queue.cc:1035] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875aeafdea04483e960993e40e2515c8" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:43:26.008745 25246 consensus_queue.cc:1035] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b712c0575245ffa417da675a79ed8a" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 34543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:43:26.021209 25249 mvcc.cc:205] Tried to move back new op lower bound from 7298345394194722816 to 7298345393732001792. Current Snapshot: MvccSnapshot[applied={T|T < 7298345394194722816}]
I20260618 22:43:26.026278 25250 mvcc.cc:205] Tried to move back new op lower bound from 7298345394194722816 to 7298345393732001792. Current Snapshot: MvccSnapshot[applied={T|T < 7298345394194722816}]
I20260618 22:43:26.037892 24769 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:43:26.038262 24769 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:43:26.042322 24769 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a641b54a809048ccafc7e93bfad28fda: Sending DeleteTablet for 3 replicas of tablet f43e035b0c2744cb88651b34cef1cfe0
I20260618 22:43:26.043578 25152 tablet_service.cc:1514] Processing DeleteTablet for tablet f43e035b0c2744cb88651b34cef1cfe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:43:26 UTC) from {username='slave'} at 127.0.0.1:55590
I20260618 22:43:26.043883 25022 tablet_service.cc:1514] Processing DeleteTablet for tablet f43e035b0c2744cb88651b34cef1cfe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:43:26 UTC) from {username='slave'} at 127.0.0.1:57676
I20260618 22:43:26.044636 24891 tablet_service.cc:1514] Processing DeleteTablet for tablet f43e035b0c2744cb88651b34cef1cfe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:43:26 UTC) from {username='slave'} at 127.0.0.1:59292
I20260618 22:43:26.048651 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 24829
I20260618 22:43:26.070565 25260 tablet_replica.cc:330] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3: stopping tablet replica
I20260618 22:43:26.075248 25260 raft_consensus.cc:2229] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:43:26.077037 25259 tablet_replica.cc:330] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8: stopping tablet replica
I20260618 22:43:26.077239 25258 tablet_replica.cc:330] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a: stopping tablet replica
I20260618 22:43:26.077618 25259 raft_consensus.cc:2229] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:43:26.077615 25258 raft_consensus.cc:2229] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:43:26.078109 25258 raft_consensus.cc:2258] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:43:26.078109 25259 raft_consensus.cc:2258] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:43:26.078488 25260 raft_consensus.cc:2258] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:43:26.078995 25233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:26.049878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59308 (local address 127.5.213.193:36179)
0618 22:43:26.051594 (+ 1716us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:26.051600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:26.070107 (+ 18507us) server_negotiation.cc:407] Connection header received
0618 22:43:26.070510 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:26.070515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:26.070657 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:26.070737 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:26.071399 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:26.072643 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:26.074654 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:26.075326 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:26.075423 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:26.077183 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:26.077194 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:26.077213 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:26.078050 (+ 837us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:26.078109 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:26.078134 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:26.078210 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:26.078462 (+ 252us) server_negotiation.cc:299] Negotiation successful
0618 22:43:26.078573 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1574}
I20260618 22:43:26.080283 24951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:26.080602 25258 ts_tablet_manager.cc:1905] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:43:26.082211 25259 ts_tablet_manager.cc:1905] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:43:26.087275 25258 ts_tablet_manager.cc:1918] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:43:26.087447 25258 log.cc:1196] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/f43e035b0c2744cb88651b34cef1cfe0
I20260618 22:43:26.087918 25258 ts_tablet_manager.cc:1939] T f43e035b0c2744cb88651b34cef1cfe0 P 92b712c0575245ffa417da675a79ed8a: Deleting consensus metadata
I20260618 22:43:26.089871 24755 catalog_manager.cc:4841] TS 92b712c0575245ffa417da675a79ed8a (127.5.213.195:34543): tablet f43e035b0c2744cb88651b34cef1cfe0 (table test-table [id=d31f01dbc102467ea6ee2e940afb1133]) successfully deleted
I20260618 22:43:26.090931 25259 ts_tablet_manager.cc:1918] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:43:26.091068 25259 log.cc:1196] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/f43e035b0c2744cb88651b34cef1cfe0
I20260618 22:43:26.091434 25259 ts_tablet_manager.cc:1939] T f43e035b0c2744cb88651b34cef1cfe0 P 875aeafdea04483e960993e40e2515c8: Deleting consensus metadata
I20260618 22:43:26.092816 24757 catalog_manager.cc:4841] TS 875aeafdea04483e960993e40e2515c8 (127.5.213.194:42069): tablet f43e035b0c2744cb88651b34cef1cfe0 (table test-table [id=d31f01dbc102467ea6ee2e940afb1133]) successfully deleted
I20260618 22:43:26.082916 25260 ts_tablet_manager.cc:1905] T f43e035b0c2744cb88651b34cef1cfe0 P 1833fc7638584bb9aea18d59621bd0e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:43:26.180368 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 24829
I20260618 22:43:26.194808 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 24960
W20260618 22:43:26.194713 24754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.213.193:36179 (error 108)
I20260618 22:43:26.196807 25264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:26.195888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:36179 (local address 127.0.0.1:59312)
0618 22:43:26.196370 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:26.196550 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:36179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":208,"threads_started":1}
W20260618 22:43:26.197026 24754 catalog_manager.cc:4584] TS 1833fc7638584bb9aea18d59621bd0e3 (127.5.213.193:36179): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f43e035b0c2744cb88651b34cef1cfe0: Network error: Client connection negotiation failed: client connection to 127.5.213.193:36179: connect: Connection refused (error 111)
I20260618 22:43:26.201738 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:26.195576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57688 (local address 127.5.213.194:42069)
0618 22:43:26.195796 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:26.195802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:26.195821 (+ 19us) server_negotiation.cc:407] Connection header received
0618 22:43:26.195980 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:26.195985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:26.196103 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:26.196223 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:26.196692 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:26.197559 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:26.198937 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:26.199370 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:26.199468 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:26.200479 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:26.200488 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:26.200501 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:26.201049 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:26.201094 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:26.201113 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:26.201197 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:26.201318 (+ 121us) server_negotiation.cc:299] Negotiation successful
0618 22:43:26.201401 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260618 22:43:26.202466 25082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:26.250932 25264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:26.250361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:36179 (local address 127.0.0.1:59314)
0618 22:43:26.250604 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:26.250786 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:36179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260618 22:43:26.252012 25264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:26.251586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:36179 (local address 127.0.0.1:59318)
0618 22:43:26.251761 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:26.251860 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:36179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260618 22:43:26.289882 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 24960
I20260618 22:43:26.303505 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 25090
I20260618 22:43:26.310496 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:26.304291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55604 (local address 127.5.213.195:34543)
0618 22:43:26.304454 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:26.304458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:26.304509 (+ 51us) server_negotiation.cc:407] Connection header received
0618 22:43:26.304719 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:26.304724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:26.304811 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:26.304894 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:26.305303 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:26.306046 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:26.307602 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:26.308000 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:26.308094 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:26.309169 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:26.309180 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:26.309194 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:26.309868 (+ 674us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:26.309916 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:26.309936 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:26.310021 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:26.310138 (+ 117us) server_negotiation.cc:299] Negotiation successful
0618 22:43:26.310225 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 22:43:26.311122 25212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:26.312510 25264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:26.311999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:36179 (local address 127.0.0.1:59334)
0618 22:43:26.312220 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:26.312334 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:36179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260618 22:43:26.313491 25264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:26.313141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:36179 (local address 127.0.0.1:59348)
0618 22:43:26.313293 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:26.313359 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:36179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260618 22:43:26.394011 25264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:26.393518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:36179 (local address 127.0.0.1:59354)
0618 22:43:26.393771 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:26.393870 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:36179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260618 22:43:26.395143 25264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:26.394738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:36179 (local address 127.0.0.1:59356)
0618 22:43:26.394926 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:26.394997 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.193:36179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260618 22:43:26.398722 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 25090
I20260618 22:43:26.411384 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 24739
I20260618 22:43:26.412425 24800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:26.522814 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 24739
2026-06-18T22:43:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2949 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:43:26 dist-test-slave-lnj3 krb5kdc[25270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:43:26 dist-test-slave-lnj3 krb5kdc[25270](info): set up 2 sockets
Jun 18 22:43:26 dist-test-slave-lnj3 krb5kdc[25270](info): commencing operation
krb5kdc: starting...
W20260618 22:43:27.554708 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.009s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:43:27 dist-test-slave-lnj3 krb5kdc[25270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:43:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:43:27.790145 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:32863
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:32863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:28.109205 25283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:28.109476 25283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:28.109587 25283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:28.116902 25283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:43:28.117019 25283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:28.117076 25283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:28.117154 25283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:43:28.117213 25283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:43:28.127300 25283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:32863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:32863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:28.128621 25283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:28.129994 25283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:28.141378 25288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:28.141337 25291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:28.141407 25289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:28.142544 25283 server_base.cc:1034] running on GCE node
I20260618 22:43:28.143332 25283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:28.144687 25283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:28.145898 25283 hybrid_clock.cc:648] HybridClock initialized: now 1781822608145840 us; error 45 us; skew 500 ppm
I20260618 22:43:28.146229 25283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:28 dist-test-slave-lnj3 krb5kdc[25270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822608, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:28.153707 25283 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:43:28.155287 25283 webserver.cc:460] Webserver started at http://127.5.213.254:40059/ using document root <none> and password file <none>
I20260618 22:43:28.155985 25283 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:28.156104 25283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:28.156411 25283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:28.159418 25283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "0b4c50d9b0824c10bd49f7fd27843950"
format_stamp: "Formatted at 2026-06-18 22:43:28 on dist-test-slave-lnj3"
server_key: "72f80ef187a4ac9ed416412fc0e5a8b6"
server_key_iv: "2d055538f4e1820c91850e44a4ea3026"
server_key_version: "encryptionkey@0"
I20260618 22:43:28.160151 25283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "0b4c50d9b0824c10bd49f7fd27843950"
format_stamp: "Formatted at 2026-06-18 22:43:28 on dist-test-slave-lnj3"
server_key: "72f80ef187a4ac9ed416412fc0e5a8b6"
server_key_iv: "2d055538f4e1820c91850e44a4ea3026"
server_key_version: "encryptionkey@0"
I20260618 22:43:28.165215 25283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:43:28.168542 25298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:28.169798 25283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:43:28.169945 25283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "0b4c50d9b0824c10bd49f7fd27843950"
format_stamp: "Formatted at 2026-06-18 22:43:28 on dist-test-slave-lnj3"
server_key: "72f80ef187a4ac9ed416412fc0e5a8b6"
server_key_iv: "2d055538f4e1820c91850e44a4ea3026"
server_key_version: "encryptionkey@0"
I20260618 22:43:28.170111 25283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:28.185556 25283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:28.189390 25283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:28.189639 25283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:28.207134 25283 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:32863
I20260618 22:43:28.207141 25349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:32863 every 8 connection(s)
I20260618 22:43:28.208521 25283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:43:28.212340 25350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:28.212639 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 25283
I20260618 22:43:28.212797 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:43:28.213188 5975 external_mini_cluster.cc:1428] Setting key 58d224dbad8e86b4fe3c6b05eacf829c
I20260618 22:43:28.226933 25350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950: Bootstrap starting.
Jun 18 22:43:28 dist-test-slave-lnj3 krb5kdc[25270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:43:28.230140 25350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:28.231457 25350 log.cc:824] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:28.234892 25350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950: No bootstrap required, opened a new log
I20260618 22:43:28.239279 25353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:28.215083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55730 (local address 127.5.213.254:32863)
0618 22:43:28.216000 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:28.216016 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:28.216125 (+ 109us) server_negotiation.cc:407] Connection header received
0618 22:43:28.219006 (+ 2881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:28.219041 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:28.219673 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:28.220336 (+ 663us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:28.221741 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:28.223757 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:28.225162 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:28.226180 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:28.226382 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:28.230561 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:28.230572 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:28.230593 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:28.230666 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:28.234415 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:28.235199 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:28.235207 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:28.235224 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:28.235450 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:28.235830 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:28.235835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:28.235838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:28.236654 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:28.236947 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:28.237375 (+ 428us) server_negotiation.cc:299] Negotiation successful
0618 22:43:28.237835 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":354,"threads_started":1}
I20260618 22:43:28.243954 25350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4c50d9b0824c10bd49f7fd27843950" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 32863 } }
I20260618 22:43:28.244300 25350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:28.244372 25350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b4c50d9b0824c10bd49f7fd27843950, State: Initialized, Role: FOLLOWER
I20260618 22:43:28.244904 25350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4c50d9b0824c10bd49f7fd27843950" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 32863 } }
I20260618 22:43:28.245116 25350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:43:28.245214 25350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:43:28.245330 25350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:28.247493 25350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4c50d9b0824c10bd49f7fd27843950" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 32863 } }
I20260618 22:43:28.248039 25350 leader_election.cc:302] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b4c50d9b0824c10bd49f7fd27843950; no voters:
I20260618 22:43:28.248634 25350 leader_election.cc:288] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:43:28.248734 25355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:28.249363 25355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950 [term 1 LEADER]: Becoming Leader. State: Replica: 0b4c50d9b0824c10bd49f7fd27843950, State: Running, Role: LEADER
I20260618 22:43:28.249953 25355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4c50d9b0824c10bd49f7fd27843950" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 32863 } }
I20260618 22:43:28.250880 25350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:43:28.252944 25356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b4c50d9b0824c10bd49f7fd27843950" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4c50d9b0824c10bd49f7fd27843950" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 32863 } } }
I20260618 22:43:28.253301 25356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:28.255249 25362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:43:28.255271 25357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b4c50d9b0824c10bd49f7fd27843950. Latest consensus state: current_term: 1 leader_uuid: "0b4c50d9b0824c10bd49f7fd27843950" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4c50d9b0824c10bd49f7fd27843950" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 32863 } } }
I20260618 22:43:28.255661 25357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:28.260355 25362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:43:28.268829 25362 catalog_manager.cc:1347] Generated new cluster ID: a3863e3851ad4ca28b883f2263984edb
I20260618 22:43:28.268989 25362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:43:28.289647 25362 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:43:28.291145 25362 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:43:28.309363 25362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950: Generated new TSK 0
I20260618 22:43:28.310267 25362 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
Entry for principal kudu/127.5.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab.
I20260618 22:43:28.373394 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:32863
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:28.692621 25376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:28.692987 25376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:28.693142 25376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:28.700246 25376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:28.700364 25376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:28.700520 25376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:43:28.709793 25376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.193.keytab
--principal=kudu/127.5.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:32863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:28.711043 25376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:28.712316 25376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:28.724668 25381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:28.724686 25384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:28.724625 25382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:28.725508 25376 server_base.cc:1034] running on GCE node
I20260618 22:43:28.726048 25376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:28.726948 25376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:28.728168 25376 hybrid_clock.cc:648] HybridClock initialized: now 1781822608728094 us; error 42 us; skew 500 ppm
I20260618 22:43:28.728528 25376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:28 dist-test-slave-lnj3 krb5kdc[25270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822608, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:28.731510 25376 init.cc:377] Logged in from keytab as kudu/127.5.213.193@KRBTEST.COM (short username kudu)
I20260618 22:43:28.732928 25376 webserver.cc:460] Webserver started at http://127.5.213.193:39343/ using document root <none> and password file <none>
I20260618 22:43:28.733564 25376 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:28.733681 25376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:28.733965 25376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:28.736827 25376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "f820017ab237440384de12fead5409ec"
format_stamp: "Formatted at 2026-06-18 22:43:28 on dist-test-slave-lnj3"
server_key: "a1f3e6cc706799b806ab0dea2ea6457b"
server_key_iv: "95514d6f400aedd2898e58ae5cd7efef"
server_key_version: "encryptionkey@0"
I20260618 22:43:28.737677 25376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "f820017ab237440384de12fead5409ec"
format_stamp: "Formatted at 2026-06-18 22:43:28 on dist-test-slave-lnj3"
server_key: "a1f3e6cc706799b806ab0dea2ea6457b"
server_key_iv: "95514d6f400aedd2898e58ae5cd7efef"
server_key_version: "encryptionkey@0"
I20260618 22:43:28.742476 25376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260618 22:43:28.745651 25391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:28.747010 25376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:43:28.747171 25376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "f820017ab237440384de12fead5409ec"
format_stamp: "Formatted at 2026-06-18 22:43:28 on dist-test-slave-lnj3"
server_key: "a1f3e6cc706799b806ab0dea2ea6457b"
server_key_iv: "95514d6f400aedd2898e58ae5cd7efef"
server_key_version: "encryptionkey@0"
I20260618 22:43:28.747344 25376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:28.761281 25376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:28.764940 25376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:28.765208 25376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:28.766048 25376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:43:28.767313 25376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:43:28.767385 25376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:28.767503 25376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:43:28.767557 25376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:28.796535 25376 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:36685
I20260618 22:43:28.796573 25503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:36685 every 8 connection(s)
I20260618 22:43:28.797839 25376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:43:28.799134 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 25376
I20260618 22:43:28.799319 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:43:28.799778 5975 external_mini_cluster.cc:1428] Setting key 8bd9cce65a4db3922c8127c0048c6f51
Jun 18 22:43:28 dist-test-slave-lnj3 krb5kdc[25270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822608, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.193@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:43:28.818874 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:28.800442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:42707 (local address 127.5.213.254:32863)
0618 22:43:28.800880 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:28.800886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:28.802131 (+ 1245us) server_negotiation.cc:407] Connection header received
0618 22:43:28.804392 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:28.804396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:28.804462 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:28.804554 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:28.806478 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:28.807234 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:28.809393 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:28.809822 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:28.809915 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:28.813592 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:28.813597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:28.813600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:28.813636 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:28.816000 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:28.816708 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:28.816713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:28.816715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:28.816832 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:28.817220 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:28.817225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:28.817228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:28.817497 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:28.817623 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:28.818279 (+ 656us) server_negotiation.cc:299] Negotiation successful
0618 22:43:28.818502 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":212,"threads_started":1}
I20260618 22:43:28.819881 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:28.800996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:32863 (local address 127.5.213.193:42707)
0618 22:43:28.801748 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:28.801807 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:28.804075 (+ 2268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:28.804808 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:28.804820 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:28.805527 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:43:28.806306 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:28.806330 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:28.807365 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:28.807370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:28.809256 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:28.809271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:28.810033 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:28.810039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:28.810309 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:28.811021 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:43:28.811050 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:43:28.813376 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:43:28.816137 (+ 2761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:43:28.816146 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:43:28.816166 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:43:28.816589 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:43:28.816930 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:43:28.816935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:43:28.816939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:43:28.817122 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:43:28.817622 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:43:28.817630 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:43:28.817952 (+ 322us) client_negotiation.cc:769] Sending connection context
0618 22:43:28.818185 (+ 233us) client_negotiation.cc:240] Negotiation successful
0618 22:43:28.818769 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":297,"threads_started":1}
I20260618 22:43:28.821425 25504 heartbeater.cc:347] Connected to a master server at 127.5.213.254:32863
I20260618 22:43:28.821733 25504 heartbeater.cc:464] Registering TS with master...
I20260618 22:43:28.822445 25504 heartbeater.cc:511] Master 127.5.213.254:32863 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.213.194@KRBTEST.COM; defaulting to no policy
I20260618 22:43:28.824642 25315 ts_manager.cc:194] Registered new tserver with Master: f820017ab237440384de12fead5409ec (127.5.213.193:36685)
I20260618 22:43:28.826122 25315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.193@KRBTEST.COM'} at 127.5.213.193:42707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
Entry for principal kudu/127.5.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab.
I20260618 22:43:28.853251 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:32863
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:29.162685 25510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:29.162977 25510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:29.163075 25510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:29.170253 25510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:29.170363 25510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:29.170472 25510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:43:29.180187 25510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.194.keytab
--principal=kudu/127.5.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:32863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:29.181408 25510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:29.182619 25510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:29.195322 25515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:29.195307 25516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:29.195353 25518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:29.196198 25510 server_base.cc:1034] running on GCE node
I20260618 22:43:29.196774 25510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:29.197736 25510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:29.198931 25510 hybrid_clock.cc:648] HybridClock initialized: now 1781822609198882 us; error 39 us; skew 500 ppm
I20260618 22:43:29.199247 25510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:29 dist-test-slave-lnj3 krb5kdc[25270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822609, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:29.202207 25510 init.cc:377] Logged in from keytab as kudu/127.5.213.194@KRBTEST.COM (short username kudu)
I20260618 22:43:29.203641 25510 webserver.cc:460] Webserver started at http://127.5.213.194:39313/ using document root <none> and password file <none>
I20260618 22:43:29.204294 25510 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:29.204404 25510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:29.204685 25510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:29.207568 25510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "268f34b6e28043a2b76f1d3de9b69279"
format_stamp: "Formatted at 2026-06-18 22:43:29 on dist-test-slave-lnj3"
server_key: "cd4e4fb1b4ab2f814d96efd6d5da86cf"
server_key_iv: "6822e7ee686b45075582e92e9fd28711"
server_key_version: "encryptionkey@0"
I20260618 22:43:29.208290 25510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "268f34b6e28043a2b76f1d3de9b69279"
format_stamp: "Formatted at 2026-06-18 22:43:29 on dist-test-slave-lnj3"
server_key: "cd4e4fb1b4ab2f814d96efd6d5da86cf"
server_key_iv: "6822e7ee686b45075582e92e9fd28711"
server_key_version: "encryptionkey@0"
I20260618 22:43:29.213239 25510 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:43:29.216729 25525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:29.218060 25510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:43:29.218214 25510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "268f34b6e28043a2b76f1d3de9b69279"
format_stamp: "Formatted at 2026-06-18 22:43:29 on dist-test-slave-lnj3"
server_key: "cd4e4fb1b4ab2f814d96efd6d5da86cf"
server_key_iv: "6822e7ee686b45075582e92e9fd28711"
server_key_version: "encryptionkey@0"
I20260618 22:43:29.218396 25510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:29.236739 25510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:29.240638 25510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:29.240906 25510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:29.241896 25510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:43:29.243216 25510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:43:29.243297 25510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:29.243415 25510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:43:29.243468 25510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:29.272832 25510 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:46863
I20260618 22:43:29.272877 25637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:46863 every 8 connection(s)
I20260618 22:43:29.274060 25510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:43:29.274742 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 25510
I20260618 22:43:29.274931 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:43:29.275374 5975 external_mini_cluster.cc:1428] Setting key e764659b9e8105ab67bcc5fcfff0ace5
Jun 18 22:43:29 dist-test-slave-lnj3 krb5kdc[25270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822609, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.194@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:43:29.294793 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:29.276760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:35867 (local address 127.5.213.254:32863)
0618 22:43:29.276961 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:29.276967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:29.278244 (+ 1277us) server_negotiation.cc:407] Connection header received
0618 22:43:29.280203 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:29.280208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:29.280293 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:29.280378 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:29.281960 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.282725 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.284300 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.284702 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.284802 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:29.288956 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:29.288961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:29.288965 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:29.288999 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:29.291544 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:29.292436 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:29.292440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:29.292443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:29.292585 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:29.293025 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:29.293028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:29.293031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:29.293312 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:29.293426 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:29.294314 (+ 888us) server_negotiation.cc:299] Negotiation successful
0618 22:43:29.294492 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260618 22:43:29.296236 25640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:29.277348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:32863 (local address 127.5.213.194:35867)
0618 22:43:29.278085 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:29.278126 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:29.279927 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:29.280657 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:29.280666 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:29.281192 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:43:29.281787 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:29.281804 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.282872 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.282876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:29.284143 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:29.284155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.284929 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.284936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:29.285255 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:29.285968 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:43:29.286001 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:43:29.288711 (+ 2710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:43:29.291688 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:43:29.291698 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:43:29.291715 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:43:29.292291 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:43:29.292691 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:43:29.292695 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:43:29.292698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:43:29.292929 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:43:29.293426 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:43:29.293433 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:43:29.293858 (+ 425us) client_negotiation.cc:769] Sending connection context
0618 22:43:29.294187 (+ 329us) client_negotiation.cc:240] Negotiation successful
0618 22:43:29.294793 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":301,"threads_started":1}
I20260618 22:43:29.297833 25638 heartbeater.cc:347] Connected to a master server at 127.5.213.254:32863
I20260618 22:43:29.298120 25638 heartbeater.cc:464] Registering TS with master...
I20260618 22:43:29.298887 25638 heartbeater.cc:511] Master 127.5.213.254:32863 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.213.195@KRBTEST.COM; defaulting to no policy
I20260618 22:43:29.300354 25315 ts_manager.cc:194] Registered new tserver with Master: 268f34b6e28043a2b76f1d3de9b69279 (127.5.213.194:46863)
I20260618 22:43:29.301183 25315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.194@KRBTEST.COM'} at 127.5.213.194:35867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
Entry for principal kudu/127.5.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab.
I20260618 22:43:29.329286 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:32863
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:37005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:29.647459 25643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:29.647749 25643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:29.647886 25643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:29.655139 25643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:29.655246 25643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:29.655387 25643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:43:29.664966 25643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:37005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.195.keytab
--principal=kudu/127.5.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:32863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:29.666260 25643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:29.667550 25643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:29.680166 25651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:29.680209 25648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:29.680214 25649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:29.681329 25643 server_base.cc:1034] running on GCE node
I20260618 22:43:29.681933 25643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:29.682936 25643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:29.684147 25643 hybrid_clock.cc:648] HybridClock initialized: now 1781822609684077 us; error 57 us; skew 500 ppm
I20260618 22:43:29.684449 25643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:29 dist-test-slave-lnj3 krb5kdc[25270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822609, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:29.687423 25643 init.cc:377] Logged in from keytab as kudu/127.5.213.195@KRBTEST.COM (short username kudu)
I20260618 22:43:29.688778 25643 webserver.cc:460] Webserver started at http://127.5.213.195:33229/ using document root <none> and password file <none>
I20260618 22:43:29.689420 25643 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:29.689531 25643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:29.689814 25643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:29.692597 25643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "e9957636c47543e3806ce765d7ba2d69"
format_stamp: "Formatted at 2026-06-18 22:43:29 on dist-test-slave-lnj3"
server_key: "31d15cf3c8597222b8308d662656669f"
server_key_iv: "07f7c3b057233dc34837d285f4d3e0c5"
server_key_version: "encryptionkey@0"
I20260618 22:43:29.693334 25643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "e9957636c47543e3806ce765d7ba2d69"
format_stamp: "Formatted at 2026-06-18 22:43:29 on dist-test-slave-lnj3"
server_key: "31d15cf3c8597222b8308d662656669f"
server_key_iv: "07f7c3b057233dc34837d285f4d3e0c5"
server_key_version: "encryptionkey@0"
I20260618 22:43:29.698069 25643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260618 22:43:29.701342 25658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:29.702579 25643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:43:29.702737 25643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "e9957636c47543e3806ce765d7ba2d69"
format_stamp: "Formatted at 2026-06-18 22:43:29 on dist-test-slave-lnj3"
server_key: "31d15cf3c8597222b8308d662656669f"
server_key_iv: "07f7c3b057233dc34837d285f4d3e0c5"
server_key_version: "encryptionkey@0"
I20260618 22:43:29.702903 25643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:29.753047 25643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:29.757257 25643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:29.757498 25643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:29.758424 25643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:43:29.759756 25643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:43:29.759835 25643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:29.759930 25643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:43:29.759971 25643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:29.788708 25643 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:44855
I20260618 22:43:29.788740 25770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:44855 every 8 connection(s)
I20260618 22:43:29.789914 25643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:43:29.792132 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 25643
I20260618 22:43:29.792299 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:43:29.792712 5975 external_mini_cluster.cc:1428] Setting key 1bfb76d9e2735808921aa74c0c7c4cb5
Jun 18 22:43:29 dist-test-slave-lnj3 krb5kdc[25270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822609, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.195@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:43:29.809993 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:29.793522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:54067 (local address 127.5.213.254:32863)
0618 22:43:29.793836 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:29.793841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:29.793986 (+ 145us) server_negotiation.cc:407] Connection header received
0618 22:43:29.796170 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:29.796176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:29.796258 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:29.796369 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:29.797968 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.799081 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.800667 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.801263 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.801363 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:29.804770 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:29.804780 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:29.804783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:29.804818 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:29.807416 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:29.808136 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:29.808155 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:29.808158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:29.808277 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:29.808603 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:29.808606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:29.808608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:29.808867 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:29.808973 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:29.809589 (+ 616us) server_negotiation.cc:299] Negotiation successful
0618 22:43:29.809757 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260618 22:43:29.810985 25773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:29.793035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:32863 (local address 127.5.213.195:54067)
0618 22:43:29.793843 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:29.793888 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:29.795915 (+ 2027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:29.796559 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:29.796568 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:29.797135 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:43:29.797810 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:29.797830 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.799241 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.799245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:29.800528 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:29.800541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.801483 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.801488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:29.801726 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:29.802319 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:43:29.802346 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:43:29.804558 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:43:29.807527 (+ 2969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:43:29.807536 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:43:29.807552 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:43:29.808045 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:43:29.808357 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:43:29.808360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:43:29.808362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:43:29.808529 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:43:29.808955 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:43:29.808960 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:43:29.809289 (+ 329us) client_negotiation.cc:769] Sending connection context
0618 22:43:29.809524 (+ 235us) client_negotiation.cc:240] Negotiation successful
0618 22:43:29.809909 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":323,"threads_started":1}
I20260618 22:43:29.812233 25771 heartbeater.cc:347] Connected to a master server at 127.5.213.254:32863
I20260618 22:43:29.812505 25771 heartbeater.cc:464] Registering TS with master...
I20260618 22:43:29.813165 25771 heartbeater.cc:511] Master 127.5.213.254:32863 requested a full tablet report, sending...
I20260618 22:43:29.814535 25315 ts_manager.cc:194] Registered new tserver with Master: e9957636c47543e3806ce765d7ba2d69 (127.5.213.195:44855)
I20260618 22:43:29.815362 25315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.213.195@KRBTEST.COM'} at 127.5.213.195:54067
I20260618 22:43:29.819854 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:43:29.828868 25504 heartbeater.cc:503] Master 127.5.213.254:32863 was elected leader, sending a full tablet report...
I20260618 22:43:29.835669 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:29.823683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46816 (local address 127.5.213.254:32863)
0618 22:43:29.823835 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:29.823842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:29.824229 (+ 387us) server_negotiation.cc:407] Connection header received
0618 22:43:29.824386 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:29.824391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:29.824450 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:29.824526 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:29.825850 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.826555 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.827839 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.828235 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.828341 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:29.830893 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:29.830897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:29.830900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:29.830931 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:29.833120 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:29.833731 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:29.833736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:29.833740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:29.833861 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:29.834200 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:29.834205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:29.834209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:29.834555 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:29.834811 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:29.835173 (+ 362us) server_negotiation.cc:299] Negotiation successful
0618 22:43:29.835383 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:43:29.839349 25315 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:43:29.845968 25315 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:43:29.867709 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:29.859556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53834 (local address 127.5.213.193:36685)
0618 22:43:29.860191 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:29.860198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:29.860221 (+ 23us) server_negotiation.cc:407] Connection header received
0618 22:43:29.860314 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:29.860322 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:29.860567 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:29.861828 (+ 1261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:29.862093 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.863114 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.865552 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.866573 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.866747 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:29.866978 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:29.867218 (+ 240us) server_negotiation.cc:299] Negotiation successful
0618 22:43:29.867382 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":289,"threads_started":1}
I20260618 22:43:29.868064 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:29.859281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:36685 (local address 127.0.0.1:53834)
0618 22:43:29.859771 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:29.859793 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:29.859891 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:29.860824 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:29.860828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:29.860865 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:29.861572 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:29.861585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.863280 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.863285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:29.865373 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:29.865386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.866789 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.866796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:29.866977 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:29.867026 (+ 49us) client_negotiation.cc:769] Sending connection context
0618 22:43:29.867169 (+ 143us) client_negotiation.cc:240] Negotiation successful
0618 22:43:29.867360 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":29,"spinlock_wait_cycles":16896,"thread_start_us":253,"threads_started":1}
I20260618 22:43:29.868201 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:29.857811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:46863 (local address 127.0.0.1:42152)
0618 22:43:29.858766 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:29.858811 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:29.858963 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:29.860808 (+ 1845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:29.860816 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:29.860858 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:29.861330 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:29.861344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.863226 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.863231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:29.865666 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:29.865681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.867387 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.867392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:29.867580 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:29.867622 (+ 42us) client_negotiation.cc:769] Sending connection context
0618 22:43:29.867701 (+ 79us) client_negotiation.cc:240] Negotiation successful
0618 22:43:29.867839 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":115,"spinlock_wait_cycles":2176,"thread_start_us":519,"threads_started":1}
I20260618 22:43:29.868588 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:29.859834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:44855 (local address 127.0.0.1:41188)
0618 22:43:29.860306 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:29.860326 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:29.860424 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:29.861493 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:29.861496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:29.861524 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:29.861949 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:29.861960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.864075 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.864081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:29.866219 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:29.866232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.867893 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.867899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:29.868062 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:29.868085 (+ 23us) client_negotiation.cc:769] Sending connection context
0618 22:43:29.868138 (+ 53us) client_negotiation.cc:240] Negotiation successful
0618 22:43:29.868263 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":5,"thread_start_us":619,"threads_started":1}
I20260618 22:43:29.869217 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:29.858051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.5.213.194:46863)
0618 22:43:29.860184 (+ 2133us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:29.860192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:29.860221 (+ 29us) server_negotiation.cc:407] Connection header received
0618 22:43:29.860314 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:29.860322 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:29.860567 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:29.860948 (+ 381us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:29.861529 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.863084 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.865867 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.867178 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.868493 (+ 1315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:29.868704 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:29.868768 (+ 64us) server_negotiation.cc:299] Negotiation successful
0618 22:43:29.868918 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1254,"thread_start_us":1111,"threads_started":1}
I20260618 22:43:29.869633 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:29.860319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41188 (local address 127.5.213.195:44855)
0618 22:43:29.860924 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:29.860932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:29.860964 (+ 32us) server_negotiation.cc:407] Connection header received
0618 22:43:29.861058 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:29.861066 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:29.861292 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:29.862144 (+ 852us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:29.862396 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.863904 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.866383 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.867667 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.868882 (+ 1215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:29.869095 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:29.869170 (+ 75us) server_negotiation.cc:299] Negotiation successful
0618 22:43:29.869317 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":342,"threads_started":1}
I20260618 22:43:29.871966 25439 tablet_service.cc:1467] Processing CreateTablet for tablet ec6b2a00e5ee4cafaf002d5f09bb8ebf (DEFAULT_TABLE table=test-table [id=2f9a375e2aa247c58992ba54928c6f10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:43:29.872421 25573 tablet_service.cc:1467] Processing CreateTablet for tablet ec6b2a00e5ee4cafaf002d5f09bb8ebf (DEFAULT_TABLE table=test-table [id=2f9a375e2aa247c58992ba54928c6f10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:43:29.873263 25439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6b2a00e5ee4cafaf002d5f09bb8ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:29.873565 25573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6b2a00e5ee4cafaf002d5f09bb8ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:29.881301 25706 tablet_service.cc:1467] Processing CreateTablet for tablet ec6b2a00e5ee4cafaf002d5f09bb8ebf (DEFAULT_TABLE table=test-table [id=2f9a375e2aa247c58992ba54928c6f10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:43:29.882508 25706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6b2a00e5ee4cafaf002d5f09bb8ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:29.885685 25789 tablet_bootstrap.cc:492] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279: Bootstrap starting.
I20260618 22:43:29.887375 25790 tablet_bootstrap.cc:492] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec: Bootstrap starting.
I20260618 22:43:29.889582 25789 tablet_bootstrap.cc:654] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:29.890515 25790 tablet_bootstrap.cc:654] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:29.890852 25789 log.cc:824] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:29.891511 25790 log.cc:824] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:29.893885 25789 tablet_bootstrap.cc:492] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279: No bootstrap required, opened a new log
I20260618 22:43:29.894161 25789 ts_tablet_manager.cc:1397] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260618 22:43:29.896070 25790 tablet_bootstrap.cc:492] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec: No bootstrap required, opened a new log
I20260618 22:43:29.896437 25790 ts_tablet_manager.cc:1397] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260618 22:43:29.897030 25793 tablet_bootstrap.cc:492] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69: Bootstrap starting.
I20260618 22:43:29.900484 25793 tablet_bootstrap.cc:654] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:29.901678 25793 log.cc:824] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:29.901836 25789 raft_consensus.cc:348] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "268f34b6e28043a2b76f1d3de9b69279" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46863 } } peers { permanent_uuid: "f820017ab237440384de12fead5409ec" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36685 } } peers { permanent_uuid: "e9957636c47543e3806ce765d7ba2d69" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44855 } }
I20260618 22:43:29.902202 25789 raft_consensus.cc:374] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:29.902339 25789 raft_consensus.cc:729] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 268f34b6e28043a2b76f1d3de9b69279, State: Initialized, Role: FOLLOWER
I20260618 22:43:29.903045 25789 consensus_queue.cc:260] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "268f34b6e28043a2b76f1d3de9b69279" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46863 } } peers { permanent_uuid: "f820017ab237440384de12fead5409ec" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36685 } } peers { permanent_uuid: "e9957636c47543e3806ce765d7ba2d69" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44855 } }
I20260618 22:43:29.904052 25638 heartbeater.cc:503] Master 127.5.213.254:32863 was elected leader, sending a full tablet report...
I20260618 22:43:29.904484 25789 ts_tablet_manager.cc:1428] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260618 22:43:29.904759 25793 tablet_bootstrap.cc:492] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69: No bootstrap required, opened a new log
I20260618 22:43:29.905009 25793 ts_tablet_manager.cc:1397] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260618 22:43:29.906869 25790 raft_consensus.cc:348] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "268f34b6e28043a2b76f1d3de9b69279" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46863 } } peers { permanent_uuid: "f820017ab237440384de12fead5409ec" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36685 } } peers { permanent_uuid: "e9957636c47543e3806ce765d7ba2d69" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44855 } }
I20260618 22:43:29.907368 25790 raft_consensus.cc:374] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:29.907498 25790 raft_consensus.cc:729] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f820017ab237440384de12fead5409ec, State: Initialized, Role: FOLLOWER
I20260618 22:43:29.908214 25790 consensus_queue.cc:260] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "268f34b6e28043a2b76f1d3de9b69279" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46863 } } peers { permanent_uuid: "f820017ab237440384de12fead5409ec" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36685 } } peers { permanent_uuid: "e9957636c47543e3806ce765d7ba2d69" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44855 } }
I20260618 22:43:29.911017 25790 ts_tablet_manager.cc:1428] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260618 22:43:29.914398 25793 raft_consensus.cc:348] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "268f34b6e28043a2b76f1d3de9b69279" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46863 } } peers { permanent_uuid: "f820017ab237440384de12fead5409ec" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36685 } } peers { permanent_uuid: "e9957636c47543e3806ce765d7ba2d69" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44855 } }
I20260618 22:43:29.914806 25793 raft_consensus.cc:374] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:29.914896 25793 raft_consensus.cc:729] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9957636c47543e3806ce765d7ba2d69, State: Initialized, Role: FOLLOWER
I20260618 22:43:29.915409 25793 consensus_queue.cc:260] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "268f34b6e28043a2b76f1d3de9b69279" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46863 } } peers { permanent_uuid: "f820017ab237440384de12fead5409ec" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36685 } } peers { permanent_uuid: "e9957636c47543e3806ce765d7ba2d69" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44855 } }
I20260618 22:43:29.916373 25771 heartbeater.cc:503] Master 127.5.213.254:32863 was elected leader, sending a full tablet report...
I20260618 22:43:29.916760 25793 ts_tablet_manager.cc:1428] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260618 22:43:29.942737 25797 raft_consensus.cc:482] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:43:29.943004 25797 raft_consensus.cc:504] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "268f34b6e28043a2b76f1d3de9b69279" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46863 } } peers { permanent_uuid: "f820017ab237440384de12fead5409ec" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36685 } } peers { permanent_uuid: "e9957636c47543e3806ce765d7ba2d69" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44855 } }
I20260618 22:43:29.944633 25797 leader_election.cc:288] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 268f34b6e28043a2b76f1d3de9b69279 (127.5.213.194:46863), f820017ab237440384de12fead5409ec (127.5.213.193:36685)
I20260618 22:43:29.951376 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:29.945098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:45501 (local address 127.5.213.194:46863)
0618 22:43:29.945293 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:29.945299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:29.945319 (+ 20us) server_negotiation.cc:407] Connection header received
0618 22:43:29.945551 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:29.945556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:29.945633 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:29.945729 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:29.946275 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.947224 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.949265 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.950410 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.950555 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:29.950660 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:29.950974 (+ 314us) server_negotiation.cc:299] Negotiation successful
0618 22:43:29.951107 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260618 22:43:29.951406 25773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:29.944999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:46863 (local address 127.5.213.195:45501)
0618 22:43:29.945233 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:29.945256 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:29.945378 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:29.945909 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:29.945918 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:29.945943 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:29.946110 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:29.946121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.947348 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.947354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:29.949085 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:29.949101 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.950549 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.950555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:29.950815 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:29.950842 (+ 27us) client_negotiation.cc:769] Sending connection context
0618 22:43:29.950923 (+ 81us) client_negotiation.cc:240] Negotiation successful
0618 22:43:29.951135 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":56}
I20260618 22:43:29.952123 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:29.945674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:59931 (local address 127.5.213.193:36685)
0618 22:43:29.945869 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:29.945874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:29.946109 (+ 235us) server_negotiation.cc:407] Connection header received
0618 22:43:29.946251 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:29.946256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:29.946313 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:29.946377 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:29.947008 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.948030 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.950215 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.951256 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.951411 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:29.951480 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:29.951765 (+ 285us) server_negotiation.cc:299] Negotiation successful
0618 22:43:29.951891 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260618 22:43:29.952320 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:29.945375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:36685 (local address 127.5.213.195:59931)
0618 22:43:29.946019 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:29.946032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:29.946139 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:29.946433 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:29.946437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:29.946461 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:29.946852 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:29.946864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.948219 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.948223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:29.950069 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:29.950085 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.951441 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.951448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:29.951620 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:29.951646 (+ 26us) client_negotiation.cc:769] Sending connection context
0618 22:43:29.951722 (+ 76us) client_negotiation.cc:240] Negotiation successful
0618 22:43:29.951843 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":42,"thread_start_us":392,"threads_started":1}
I20260618 22:43:29.952636 25593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6b2a00e5ee4cafaf002d5f09bb8ebf" candidate_uuid: "e9957636c47543e3806ce765d7ba2d69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "268f34b6e28043a2b76f1d3de9b69279" is_pre_election: true
I20260618 22:43:29.953188 25593 raft_consensus.cc:2454] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9957636c47543e3806ce765d7ba2d69 in term 0.
I20260618 22:43:29.953265 25459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6b2a00e5ee4cafaf002d5f09bb8ebf" candidate_uuid: "e9957636c47543e3806ce765d7ba2d69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f820017ab237440384de12fead5409ec" is_pre_election: true
I20260618 22:43:29.953835 25459 raft_consensus.cc:2454] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9957636c47543e3806ce765d7ba2d69 in term 0.
I20260618 22:43:29.953894 25659 leader_election.cc:302] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 268f34b6e28043a2b76f1d3de9b69279, e9957636c47543e3806ce765d7ba2d69; no voters:
I20260618 22:43:29.954317 25797 raft_consensus.cc:2781] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:43:29.954468 25797 raft_consensus.cc:482] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:43:29.954574 25797 raft_consensus.cc:3037] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:29.957013 25797 raft_consensus.cc:504] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "268f34b6e28043a2b76f1d3de9b69279" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46863 } } peers { permanent_uuid: "f820017ab237440384de12fead5409ec" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36685 } } peers { permanent_uuid: "e9957636c47543e3806ce765d7ba2d69" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44855 } }
I20260618 22:43:29.957582 25797 leader_election.cc:288] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [CANDIDATE]: Term 1 election: Requested vote from peers 268f34b6e28043a2b76f1d3de9b69279 (127.5.213.194:46863), f820017ab237440384de12fead5409ec (127.5.213.193:36685)
I20260618 22:43:29.958073 25593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6b2a00e5ee4cafaf002d5f09bb8ebf" candidate_uuid: "e9957636c47543e3806ce765d7ba2d69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "268f34b6e28043a2b76f1d3de9b69279"
I20260618 22:43:29.958137 25459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6b2a00e5ee4cafaf002d5f09bb8ebf" candidate_uuid: "e9957636c47543e3806ce765d7ba2d69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f820017ab237440384de12fead5409ec"
I20260618 22:43:29.958320 25459 raft_consensus.cc:3037] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:29.958321 25593 raft_consensus.cc:3037] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:29.961418 25593 raft_consensus.cc:2454] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9957636c47543e3806ce765d7ba2d69 in term 1.
I20260618 22:43:29.961418 25459 raft_consensus.cc:2454] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9957636c47543e3806ce765d7ba2d69 in term 1.
I20260618 22:43:29.961830 25659 leader_election.cc:302] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 268f34b6e28043a2b76f1d3de9b69279, e9957636c47543e3806ce765d7ba2d69; no voters:
I20260618 22:43:29.962087 25797 raft_consensus.cc:2781] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:29.962500 25797 raft_consensus.cc:686] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [term 1 LEADER]: Becoming Leader. State: Replica: e9957636c47543e3806ce765d7ba2d69, State: Running, Role: LEADER
I20260618 22:43:29.963076 25797 consensus_queue.cc:237] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "268f34b6e28043a2b76f1d3de9b69279" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46863 } } peers { permanent_uuid: "f820017ab237440384de12fead5409ec" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36685 } } peers { permanent_uuid: "e9957636c47543e3806ce765d7ba2d69" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44855 } }
I20260618 22:43:29.967548 25314 catalog_manager.cc:5495] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9957636c47543e3806ce765d7ba2d69 (127.5.213.195). New cstate: current_term: 1 leader_uuid: "e9957636c47543e3806ce765d7ba2d69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "268f34b6e28043a2b76f1d3de9b69279" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f820017ab237440384de12fead5409ec" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9957636c47543e3806ce765d7ba2d69" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 44855 } health_report { overall_health: HEALTHY } } }
I20260618 22:43:29.994150 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:29.988687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41194 (local address 127.5.213.195:44855)
0618 22:43:29.988889 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:29.988894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:29.988912 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:43:29.989020 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:29.989024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:29.989106 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:29.989224 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:43:29.989654 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.990410 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.992046 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:29.992437 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:29.992529 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:29.993024 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:43:29.993506 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:43:29.993639 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:29.993814 (+ 175us) server_negotiation.cc:299] Negotiation successful
0618 22:43:29.993893 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260618 22:43:30.002511 25593 raft_consensus.cc:1261] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279 [term 1 FOLLOWER]: Refusing update from remote peer e9957636c47543e3806ce765d7ba2d69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:43:30.002509 25459 raft_consensus.cc:1261] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec [term 1 FOLLOWER]: Refusing update from remote peer e9957636c47543e3806ce765d7ba2d69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:43:30.003571 25799 consensus_queue.cc:1035] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f820017ab237440384de12fead5409ec" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:43:30.003968 25797 consensus_queue.cc:1035] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "268f34b6e28043a2b76f1d3de9b69279" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:43:30.016166 25803 mvcc.cc:205] Tried to move back new op lower bound from 7298345410559340544 to 7298345410412871680. Current Snapshot: MvccSnapshot[applied={T|T < 7298345410559340544}]
I20260618 22:43:30.015971 25802 mvcc.cc:205] Tried to move back new op lower bound from 7298345410559340544 to 7298345410412871680. Current Snapshot: MvccSnapshot[applied={T|T < 7298345410559340544}]
I20260618 22:43:30.017414 25804 mvcc.cc:205] Tried to move back new op lower bound from 7298345410559340544 to 7298345410412871680. Current Snapshot: MvccSnapshot[applied={T|T < 7298345410559340544}]
I20260618 22:43:30.031076 25315 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:43:30.031345 25315 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:43:30.034935 25315 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b4c50d9b0824c10bd49f7fd27843950: Sending DeleteTablet for 3 replicas of tablet ec6b2a00e5ee4cafaf002d5f09bb8ebf
I20260618 22:43:30.036145 25573 tablet_service.cc:1514] Processing DeleteTablet for tablet ec6b2a00e5ee4cafaf002d5f09bb8ebf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:43:30 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:42152
I20260618 22:43:30.036641 25439 tablet_service.cc:1514] Processing DeleteTablet for tablet ec6b2a00e5ee4cafaf002d5f09bb8ebf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:43:30 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:53834
I20260618 22:43:30.037338 25706 tablet_service.cc:1514] Processing DeleteTablet for tablet ec6b2a00e5ee4cafaf002d5f09bb8ebf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:43:30 UTC) from {username='kudu', principal='kudu/127.5.213.254@KRBTEST.COM'} at 127.0.0.1:41188
I20260618 22:43:30.037437 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 25376
I20260618 22:43:30.047199 25814 tablet_replica.cc:330] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69: stopping tablet replica
I20260618 22:43:30.048815 25814 raft_consensus.cc:2229] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:43:30.050387 25814 raft_consensus.cc:2258] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:43:30.051909 25813 tablet_replica.cc:330] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec: stopping tablet replica
I20260618 22:43:30.052356 25813 raft_consensus.cc:2229] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:43:30.052882 25813 raft_consensus.cc:2258] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:43:30.053573 25812 tablet_replica.cc:330] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279: stopping tablet replica
I20260618 22:43:30.053761 25814 ts_tablet_manager.cc:1905] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:43:30.053995 25812 raft_consensus.cc:2229] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:43:30.054518 25812 raft_consensus.cc:2258] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:43:30.056550 25812 ts_tablet_manager.cc:1905] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:43:30.056860 25813 ts_tablet_manager.cc:1905] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 18 22:43:30 dist-test-slave-lnj3 krb5kdc[25270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.193@KRBTEST.COM
I20260618 22:43:30.063256 25812 ts_tablet_manager.cc:1918] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:43:30.063452 25812 log.cc:1196] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/ec6b2a00e5ee4cafaf002d5f09bb8ebf
I20260618 22:43:30.064059 25812 ts_tablet_manager.cc:1939] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P 268f34b6e28043a2b76f1d3de9b69279: Deleting consensus metadata
I20260618 22:43:30.065440 25813 ts_tablet_manager.cc:1918] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:43:30.066030 25299 catalog_manager.cc:4841] TS 268f34b6e28043a2b76f1d3de9b69279 (127.5.213.194:46863): tablet ec6b2a00e5ee4cafaf002d5f09bb8ebf (table test-table [id=2f9a375e2aa247c58992ba54928c6f10]) successfully deleted
I20260618 22:43:30.066792 25813 log.cc:1196] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/ec6b2a00e5ee4cafaf002d5f09bb8ebf
I20260618 22:43:30.067437 25813 ts_tablet_manager.cc:1939] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P f820017ab237440384de12fead5409ec: Deleting consensus metadata
I20260618 22:43:30.069325 25299 catalog_manager.cc:4841] TS f820017ab237440384de12fead5409ec (127.5.213.193:36685): tablet ec6b2a00e5ee4cafaf002d5f09bb8ebf (table test-table [id=2f9a375e2aa247c58992ba54928c6f10]) successfully deleted
I20260618 22:43:30.070997 25814 ts_tablet_manager.cc:1918] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:43:30.071204 25814 log.cc:1196] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/ec6b2a00e5ee4cafaf002d5f09bb8ebf
I20260618 22:43:30.071580 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:30.038707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53842 (local address 127.5.213.193:36685)
0618 22:43:30.038886 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:30.038892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:30.051255 (+ 12363us) server_negotiation.cc:407] Connection header received
0618 22:43:30.052679 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:30.052684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:30.052766 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:30.052847 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:30.055060 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:30.056141 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:30.058058 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:30.058702 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:30.058797 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:30.064315 (+ 5518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:30.064330 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:30.064349 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:30.064394 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:30.068086 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:30.068977 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:30.068986 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:30.068990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:30.069140 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:30.069578 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:30.069583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:30.069586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:30.070026 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:30.070307 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:30.070898 (+ 591us) server_negotiation.cc:299] Negotiation successful
0618 22:43:30.071182 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:43:30.071761 25814 ts_tablet_manager.cc:1939] T ec6b2a00e5ee4cafaf002d5f09bb8ebf P e9957636c47543e3806ce765d7ba2d69: Deleting consensus metadata
I20260618 22:43:30.072503 25499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:30.073669 25299 catalog_manager.cc:4841] TS e9957636c47543e3806ce765d7ba2d69 (127.5.213.195:44855): tablet ec6b2a00e5ee4cafaf002d5f09bb8ebf (table test-table [id=2f9a375e2aa247c58992ba54928c6f10]) successfully deleted
I20260618 22:43:30.162129 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 25376
I20260618 22:43:30.176496 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 25510
Jun 18 22:43:30 dist-test-slave-lnj3 krb5kdc[25270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.194@KRBTEST.COM
I20260618 22:43:30.191637 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:30.177253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42154 (local address 127.5.213.194:46863)
0618 22:43:30.177444 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:30.177449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:30.177469 (+ 20us) server_negotiation.cc:407] Connection header received
0618 22:43:30.177605 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:30.177611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:30.177681 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:30.177762 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:30.178988 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:30.179707 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:30.180919 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:30.181308 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:30.181412 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:30.185855 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:30.185871 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:30.185886 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:30.185930 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:30.188669 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:30.189488 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:30.189495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:30.189497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:30.189614 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:30.189993 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:30.189995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:30.189997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:30.190359 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:30.190577 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:30.190988 (+ 411us) server_negotiation.cc:299] Negotiation successful
0618 22:43:30.191148 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260618 22:43:30.192282 25633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:30.280820 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 25510
I20260618 22:43:30.294816 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 25643
Jun 18 22:43:30 dist-test-slave-lnj3 krb5kdc[25270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.195@KRBTEST.COM
I20260618 22:43:30.311787 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:30.295682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41206 (local address 127.5.213.195:44855)
0618 22:43:30.295890 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:30.295896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:30.295918 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:43:30.296060 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:30.296066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:30.296164 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:30.296293 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:30.297541 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:30.298886 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:30.300226 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:30.300910 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:30.301054 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:30.305540 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:30.305552 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:30.305570 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:30.305614 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:30.308765 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:30.309640 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:30.309649 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:30.309652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:30.309818 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:30.310236 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:30.310240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:30.310242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:30.310565 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:30.310833 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:30.311192 (+ 359us) server_negotiation.cc:299] Negotiation successful
0618 22:43:30.311421 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260618 22:43:30.312829 25766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:30.432706 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 25643
I20260618 22:43:30.448029 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 25283
I20260618 22:43:30.449137 25345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:30.558372 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 25283
2026-06-18T22:43:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4035 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-18T22:43:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:30Z Disabled control of system clock
I20260618 22:43:30.627415 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:37715
--webserver_interface=127.5.213.254
--webserver_port=0
--builtin_ntp_servers=127.5.213.212:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:37715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 22:43:30.945749 25823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:30.946071 25823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:30.946204 25823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:30.953361 25823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:43:30.953486 25823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:30.953552 25823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:30.953636 25823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:43:30.953694 25823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:43:30.962920 25823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:37715
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:37715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:30.964211 25823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:30.965950 25823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:30.976857 25828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:30.976899 25829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:30.976933 25831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:30.977896 25823 server_base.cc:1034] running on GCE node
I20260618 22:43:30.978749 25823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:30.980227 25823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:30.981428 25823 hybrid_clock.cc:648] HybridClock initialized: now 1781822610981387 us; error 31 us; skew 500 ppm
I20260618 22:43:30.981916 25823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:43:30.984288 25823 webserver.cc:460] Webserver started at http://127.5.213.254:41263/ using document root <none> and password file <none>
I20260618 22:43:30.984959 25823 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:30.985050 25823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:30.985349 25823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:30.988348 25823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "74780dcb3e614f66ab5f3e8be3f65b6f"
format_stamp: "Formatted at 2026-06-18 22:43:30 on dist-test-slave-lnj3"
server_key: "69f64f9a536e68a2a895db8aa1199ebd"
server_key_iv: "8ea7afce6c209b4af933080a59ea9cf4"
server_key_version: "encryptionkey@0"
I20260618 22:43:30.989151 25823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "74780dcb3e614f66ab5f3e8be3f65b6f"
format_stamp: "Formatted at 2026-06-18 22:43:30 on dist-test-slave-lnj3"
server_key: "69f64f9a536e68a2a895db8aa1199ebd"
server_key_iv: "8ea7afce6c209b4af933080a59ea9cf4"
server_key_version: "encryptionkey@0"
I20260618 22:43:30.994381 25823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260618 22:43:30.998034 25837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:30.999615 25823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:43:30.999786 25823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "74780dcb3e614f66ab5f3e8be3f65b6f"
format_stamp: "Formatted at 2026-06-18 22:43:30 on dist-test-slave-lnj3"
server_key: "69f64f9a536e68a2a895db8aa1199ebd"
server_key_iv: "8ea7afce6c209b4af933080a59ea9cf4"
server_key_version: "encryptionkey@0"
I20260618 22:43:30.999995 25823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:31.030529 25823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:31.031450 25823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:31.031730 25823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:31.049086 25823 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:37715
I20260618 22:43:31.049109 25888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:37715 every 8 connection(s)
I20260618 22:43:31.050451 25823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:43:31.054318 25889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:31.058835 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 25823
I20260618 22:43:31.059003 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:43:31.059520 5975 external_mini_cluster.cc:1428] Setting key 43dc65b07944428882bff1a08b33b497
I20260618 22:43:31.065658 25889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f: Bootstrap starting.
I20260618 22:43:31.069245 25889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:31.070437 25889 log.cc:824] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:31.073463 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:31.061433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35486 (local address 127.5.213.254:37715)
0618 22:43:31.062380 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:31.062391 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:31.062438 (+ 47us) server_negotiation.cc:407] Connection header received
0618 22:43:31.064228 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:31.064252 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:31.064722 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:31.065036 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:31.065634 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:31.067047 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:31.068872 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:31.069607 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:31.069777 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:31.070905 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:31.070913 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:31.070928 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:31.071714 (+ 786us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:31.071817 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:31.071843 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:31.071959 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:31.072190 (+ 231us) server_negotiation.cc:299] Negotiation successful
0618 22:43:31.072510 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":327,"threads_started":1}
I20260618 22:43:31.073652 25889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f: No bootstrap required, opened a new log
I20260618 22:43:31.081524 25889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74780dcb3e614f66ab5f3e8be3f65b6f" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 37715 } }
I20260618 22:43:31.081923 25889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:31.082026 25889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74780dcb3e614f66ab5f3e8be3f65b6f, State: Initialized, Role: FOLLOWER
I20260618 22:43:31.082684 25889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74780dcb3e614f66ab5f3e8be3f65b6f" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 37715 } }
I20260618 22:43:31.082921 25889 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:43:31.083055 25889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:43:31.083235 25889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:31.085402 25889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74780dcb3e614f66ab5f3e8be3f65b6f" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 37715 } }
I20260618 22:43:31.085913 25889 leader_election.cc:302] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74780dcb3e614f66ab5f3e8be3f65b6f; no voters:
I20260618 22:43:31.086477 25889 leader_election.cc:288] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:43:31.086615 25894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:31.087380 25894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f [term 1 LEADER]: Becoming Leader. State: Replica: 74780dcb3e614f66ab5f3e8be3f65b6f, State: Running, Role: LEADER
I20260618 22:43:31.087874 25894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74780dcb3e614f66ab5f3e8be3f65b6f" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 37715 } }
I20260618 22:43:31.088513 25889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:43:31.090840 25896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74780dcb3e614f66ab5f3e8be3f65b6f. Latest consensus state: current_term: 1 leader_uuid: "74780dcb3e614f66ab5f3e8be3f65b6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74780dcb3e614f66ab5f3e8be3f65b6f" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 37715 } } }
I20260618 22:43:31.090878 25895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74780dcb3e614f66ab5f3e8be3f65b6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74780dcb3e614f66ab5f3e8be3f65b6f" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 37715 } } }
I20260618 22:43:31.091250 25895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:31.091248 25896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:31.092095 25900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:43:31.098184 25900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:43:31.107429 25900 catalog_manager.cc:1347] Generated new cluster ID: 1855730066f44f6184d7064c2da8e242
I20260618 22:43:31.107563 25900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:43:31.125278 25900 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:43:31.126591 25900 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:43:31.142356 25900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f: Generated new TSK 0
I20260618 22:43:31.143541 25900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:43:31.159650 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.193:0
--local_ip_for_outbound_sockets=127.5.213.193
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:37715
--builtin_ntp_servers=127.5.213.212:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 22:43:31.479950 25913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:31.480278 25913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:31.480412 25913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:31.488721 25913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:31.488845 25913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:31.489014 25913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.193
I20260618 22:43:31.499311 25913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.213.193
--webserver_port=0
--tserver_master_addrs=127.5.213.254:37715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.193
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:31.500634 25913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:31.502374 25913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:31.514907 25919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:31.514922 25921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:31.515262 25918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:31.515890 25913 server_base.cc:1034] running on GCE node
I20260618 22:43:31.516510 25913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:31.517474 25913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:31.518734 25913 hybrid_clock.cc:648] HybridClock initialized: now 1781822611518659 us; error 49 us; skew 500 ppm
I20260618 22:43:31.519052 25913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:43:31.521409 25913 webserver.cc:460] Webserver started at http://127.5.213.193:35955/ using document root <none> and password file <none>
I20260618 22:43:31.522064 25913 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:31.522159 25913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:31.522419 25913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:31.525286 25913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-0/data/instance:
uuid: "678989c097da44258201f7814841ea7e"
format_stamp: "Formatted at 2026-06-18 22:43:31 on dist-test-slave-lnj3"
server_key: "3b65a83e0218dd9bb4a4eaafceb5254d"
server_key_iv: "d01ab1dc68d10071d0415bbb1e82b314"
server_key_version: "encryptionkey@0"
I20260618 22:43:31.526000 25913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance:
uuid: "678989c097da44258201f7814841ea7e"
format_stamp: "Formatted at 2026-06-18 22:43:31 on dist-test-slave-lnj3"
server_key: "3b65a83e0218dd9bb4a4eaafceb5254d"
server_key_iv: "d01ab1dc68d10071d0415bbb1e82b314"
server_key_version: "encryptionkey@0"
I20260618 22:43:31.531098 25913 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260618 22:43:31.534502 25927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:31.535946 25913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:43:31.536093 25913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-0/wal
uuid: "678989c097da44258201f7814841ea7e"
format_stamp: "Formatted at 2026-06-18 22:43:31 on dist-test-slave-lnj3"
server_key: "3b65a83e0218dd9bb4a4eaafceb5254d"
server_key_iv: "d01ab1dc68d10071d0415bbb1e82b314"
server_key_version: "encryptionkey@0"
I20260618 22:43:31.536288 25913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:31.553514 25913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:31.554427 25913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:31.554796 25913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:31.555800 25913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:43:31.557372 25913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:43:31.557535 25913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:31.557682 25913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:43:31.557768 25913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:31.587872 25913 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.193:45489
I20260618 22:43:31.587917 26039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.193:45489 every 8 connection(s)
I20260618 22:43:31.589202 25913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-0/data/info.pb
I20260618 22:43:31.592347 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 25913
I20260618 22:43:31.592523 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-0/wal/instance
I20260618 22:43:31.593150 5975 external_mini_cluster.cc:1428] Setting key 114f82142832f7b19e8ec085e49f0f67
I20260618 22:43:31.595942 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.194:0
--local_ip_for_outbound_sockets=127.5.213.194
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:37715
--builtin_ntp_servers=127.5.213.212:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 22:43:31.602026 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:31.591822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:39381 (local address 127.5.213.254:37715)
0618 22:43:31.592272 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:31.592276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:31.593740 (+ 1464us) server_negotiation.cc:407] Connection header received
0618 22:43:31.595673 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:31.595679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:31.595760 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:31.595850 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:31.597000 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:31.597712 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:31.599302 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:31.599764 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:31.599873 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:31.601177 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:31.601181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:31.601185 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:31.601203 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:31.601243 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:31.601255 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:31.601328 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:31.601536 (+ 208us) server_negotiation.cc:299] Negotiation successful
0618 22:43:31.601639 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":232,"threads_started":1}
I20260618 22:43:31.603091 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:31.592456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:37715 (local address 127.5.213.193:39381)
0618 22:43:31.593512 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:31.593552 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:31.595404 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:31.596139 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:31.596151 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:31.596246 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:43:31.596844 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:31.596861 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:31.597860 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:31.597864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:31.599162 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:31.599175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:31.599978 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:31.599985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:31.600215 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:31.600867 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:43:31.600869 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:43:31.600910 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:43:31.600990 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:43:31.601338 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:43:31.601350 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:43:31.601358 (+ 8us) client_negotiation.cc:769] Sending connection context
0618 22:43:31.601512 (+ 154us) client_negotiation.cc:240] Negotiation successful
0618 22:43:31.601927 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":310,"threads_started":1}
I20260618 22:43:31.604650 26040 heartbeater.cc:347] Connected to a master server at 127.5.213.254:37715
I20260618 22:43:31.604936 26040 heartbeater.cc:464] Registering TS with master...
I20260618 22:43:31.605674 26040 heartbeater.cc:511] Master 127.5.213.254:37715 requested a full tablet report, sending...
I20260618 22:43:31.607728 25854 ts_manager.cc:194] Registered new tserver with Master: 678989c097da44258201f7814841ea7e (127.5.213.193:45489)
I20260618 22:43:31.609253 25854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.213.193:39381
W20260618 22:43:31.913465 26044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:31.913755 26044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:31.913882 26044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:31.921317 26044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:31.921422 26044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:31.921559 26044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.194
I20260618 22:43:31.930791 26044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.213.194
--webserver_port=0
--tserver_master_addrs=127.5.213.254:37715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.194
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:31.932101 26044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:31.933859 26044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:31.947060 26049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:31.946988 26050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:31.947029 26052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:31.947863 26044 server_base.cc:1034] running on GCE node
I20260618 22:43:31.948437 26044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:31.949331 26044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:31.950528 26044 hybrid_clock.cc:648] HybridClock initialized: now 1781822611950477 us; error 43 us; skew 500 ppm
I20260618 22:43:31.950929 26044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:43:31.953226 26044 webserver.cc:460] Webserver started at http://127.5.213.194:40511/ using document root <none> and password file <none>
I20260618 22:43:31.953909 26044 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:31.953998 26044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:31.954306 26044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:31.957211 26044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-1/data/instance:
uuid: "b0cfcaa131a240d1b162ddbac4a89239"
format_stamp: "Formatted at 2026-06-18 22:43:31 on dist-test-slave-lnj3"
server_key: "ea0122d51bb48cf94808f0a86eb15baa"
server_key_iv: "a5e05430f9887a5b1077b812036c69ed"
server_key_version: "encryptionkey@0"
I20260618 22:43:31.957990 26044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance:
uuid: "b0cfcaa131a240d1b162ddbac4a89239"
format_stamp: "Formatted at 2026-06-18 22:43:31 on dist-test-slave-lnj3"
server_key: "ea0122d51bb48cf94808f0a86eb15baa"
server_key_iv: "a5e05430f9887a5b1077b812036c69ed"
server_key_version: "encryptionkey@0"
I20260618 22:43:31.963059 26044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260618 22:43:31.966589 26058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:31.968128 26044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:43:31.968278 26044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-1/wal
uuid: "b0cfcaa131a240d1b162ddbac4a89239"
format_stamp: "Formatted at 2026-06-18 22:43:31 on dist-test-slave-lnj3"
server_key: "ea0122d51bb48cf94808f0a86eb15baa"
server_key_iv: "a5e05430f9887a5b1077b812036c69ed"
server_key_version: "encryptionkey@0"
I20260618 22:43:31.968489 26044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:31.989267 26044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:31.990100 26044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:31.990393 26044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:31.991442 26044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:43:31.992902 26044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:43:31.993003 26044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:31.993110 26044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:43:31.993181 26044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:32.023603 26044 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.194:32899
I20260618 22:43:32.023659 26170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.194:32899 every 8 connection(s)
I20260618 22:43:32.024930 26044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-1/data/info.pb
I20260618 22:43:32.029598 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 26044
I20260618 22:43:32.029789 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-1/wal/instance
I20260618 22:43:32.030254 5975 external_mini_cluster.cc:1428] Setting key c02b08ff319ea6d36222da82449b7180
I20260618 22:43:32.032716 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.213.195:0
--local_ip_for_outbound_sockets=127.5.213.195
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:37715
--builtin_ntp_servers=127.5.213.212:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 22:43:32.037349 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.027439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:50339 (local address 127.5.213.254:37715)
0618 22:43:32.027648 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:32.027653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:32.029059 (+ 1406us) server_negotiation.cc:407] Connection header received
0618 22:43:32.030937 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:32.030942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:32.031032 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:32.031118 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:32.032211 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.032951 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.034673 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.035072 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.035194 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.036551 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:32.036556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:32.036559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:32.036577 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:32.036608 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:32.036620 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:32.036692 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:32.036922 (+ 230us) server_negotiation.cc:299] Negotiation successful
0618 22:43:32.037023 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 22:43:32.038414 26173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.027913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:37715 (local address 127.5.213.194:50339)
0618 22:43:32.028882 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:32.028922 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:32.030656 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:32.031370 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:32.031379 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:32.031474 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:43:32.032045 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:32.032062 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.033110 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.033114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:32.034500 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:32.034513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.035287 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.035294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:32.035522 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.036206 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:43:32.036209 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:43:32.036253 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:43:32.036353 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:43:32.036711 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:43:32.036722 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:43:32.036729 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 22:43:32.036899 (+ 170us) client_negotiation.cc:240] Negotiation successful
0618 22:43:32.037326 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":297,"threads_started":1}
I20260618 22:43:32.040077 26171 heartbeater.cc:347] Connected to a master server at 127.5.213.254:37715
I20260618 22:43:32.040424 26171 heartbeater.cc:464] Registering TS with master...
I20260618 22:43:32.041242 26171 heartbeater.cc:511] Master 127.5.213.254:37715 requested a full tablet report, sending...
I20260618 22:43:32.042763 25854 ts_manager.cc:194] Registered new tserver with Master: b0cfcaa131a240d1b162ddbac4a89239 (127.5.213.194:32899)
I20260618 22:43:32.043545 25854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.213.194:50339
W20260618 22:43:32.363400 26174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:32.363705 26174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:32.363824 26174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:32.371207 26174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:32.371309 26174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:32.371450 26174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.213.195
I20260618 22:43:32.380945 26174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.213.195
--webserver_port=0
--tserver_master_addrs=127.5.213.254:37715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.213.195
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:32.382185 26174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:32.383992 26174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:32.396798 26179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:32.396734 26182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:32.396763 26180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:32.397341 26174 server_base.cc:1034] running on GCE node
I20260618 22:43:32.398016 26174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:32.398993 26174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:32.400262 26174 hybrid_clock.cc:648] HybridClock initialized: now 1781822612400188 us; error 45 us; skew 500 ppm
I20260618 22:43:32.400594 26174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:43:32.402917 26174 webserver.cc:460] Webserver started at http://127.5.213.195:37317/ using document root <none> and password file <none>
I20260618 22:43:32.403627 26174 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:32.403728 26174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:32.404091 26174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:32.406993 26174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-2/data/instance:
uuid: "56266490c8834ab4b020d5b29d95f95b"
format_stamp: "Formatted at 2026-06-18 22:43:32 on dist-test-slave-lnj3"
server_key: "e33403579ac8328b37fb733225ea5419"
server_key_iv: "fd5bfe59a39b5496fd6dea5c6e770f9e"
server_key_version: "encryptionkey@0"
I20260618 22:43:32.407771 26174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance:
uuid: "56266490c8834ab4b020d5b29d95f95b"
format_stamp: "Formatted at 2026-06-18 22:43:32 on dist-test-slave-lnj3"
server_key: "e33403579ac8328b37fb733225ea5419"
server_key_iv: "fd5bfe59a39b5496fd6dea5c6e770f9e"
server_key_version: "encryptionkey@0"
I20260618 22:43:32.412575 26174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260618 22:43:32.416061 26188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:32.417461 26174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:43:32.417624 26174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-2/wal
uuid: "56266490c8834ab4b020d5b29d95f95b"
format_stamp: "Formatted at 2026-06-18 22:43:32 on dist-test-slave-lnj3"
server_key: "e33403579ac8328b37fb733225ea5419"
server_key_iv: "fd5bfe59a39b5496fd6dea5c6e770f9e"
server_key_version: "encryptionkey@0"
I20260618 22:43:32.417834 26174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:32.436352 26174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:32.437218 26174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:32.437501 26174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:32.438479 26174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:43:32.440184 26174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:43:32.440285 26174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:32.440387 26174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:43:32.440452 26174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:32.470868 26174 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.195:39187
I20260618 22:43:32.470921 26300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.195:39187 every 8 connection(s)
I20260618 22:43:32.472131 26174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-2/data/info.pb
I20260618 22:43:32.477092 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 26174
I20260618 22:43:32.477273 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-2/wal/instance
I20260618 22:43:32.477865 5975 external_mini_cluster.cc:1428] Setting key c91e297db0e218a11dd159180fc07e33
I20260618 22:43:32.485131 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.474751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.195:40879 (local address 127.5.213.254:37715)
0618 22:43:32.474948 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:32.474954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:32.476420 (+ 1466us) server_negotiation.cc:407] Connection header received
0618 22:43:32.478888 (+ 2468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:32.478895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:32.478979 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:32.479084 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:32.480390 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.481216 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.482795 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.483183 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.483279 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.484460 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:32.484465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:32.484468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:32.484485 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:32.484516 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:32.484528 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:32.484602 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:32.484797 (+ 195us) server_negotiation.cc:299] Negotiation successful
0618 22:43:32.484880 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:43:32.486059 26303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.475351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:37715 (local address 127.5.213.195:40879)
0618 22:43:32.476258 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:32.476301 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:32.478551 (+ 2250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:32.479391 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:32.479405 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:32.479525 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:43:32.480222 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:32.480241 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.481348 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.481352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:32.482653 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:32.482668 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.483352 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.483356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:32.483547 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.484152 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:43:32.484155 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:43:32.484201 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:43:32.484281 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:43:32.484611 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:43:32.484618 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:43:32.484624 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 22:43:32.484763 (+ 139us) client_negotiation.cc:240] Negotiation successful
0618 22:43:32.485048 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":268,"threads_started":1}
I20260618 22:43:32.487304 26301 heartbeater.cc:347] Connected to a master server at 127.5.213.254:37715
I20260618 22:43:32.487578 26301 heartbeater.cc:464] Registering TS with master...
I20260618 22:43:32.488363 26301 heartbeater.cc:511] Master 127.5.213.254:37715 requested a full tablet report, sending...
I20260618 22:43:32.489631 25854 ts_manager.cc:194] Registered new tserver with Master: 56266490c8834ab4b020d5b29d95f95b (127.5.213.195:39187)
I20260618 22:43:32.490406 25854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.213.195:40879
I20260618 22:43:32.492781 5975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:43:32.502257 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.496733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35488 (local address 127.5.213.254:37715)
0618 22:43:32.496877 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:32.496883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:32.497271 (+ 388us) server_negotiation.cc:407] Connection header received
0618 22:43:32.497430 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:32.497436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:32.497506 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:32.497586 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:32.498040 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.498783 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.500108 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.500502 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.500606 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.501611 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:32.501622 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:32.501625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:32.501643 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:32.501675 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:32.501687 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:32.501759 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:32.501884 (+ 125us) server_negotiation.cc:299] Negotiation successful
0618 22:43:32.501976 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:43:32.506064 25854 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:43:32.512691 25854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:43:32.548470 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.524952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55206 (local address 127.5.213.193:45489)
0618 22:43:32.539389 (+ 14437us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:32.539504 (+ 115us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:32.539534 (+ 30us) server_negotiation.cc:407] Connection header received
0618 22:43:32.539628 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:32.539634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:32.539870 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:32.540030 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:32.541239 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.542867 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.545841 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.547255 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.547373 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.547651 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:32.547942 (+ 291us) server_negotiation.cc:299] Negotiation successful
0618 22:43:32.548115 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13654,"thread_start_us":13540,"threads_started":1}
I20260618 22:43:32.548590 26313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.525006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:32899 (local address 127.0.0.1:41042)
0618 22:43:32.539003 (+ 13997us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:32.539020 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:32.539152 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:32.540301 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:32.540305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:32.540349 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:32.541052 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:32.541062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.542987 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.542993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:32.545625 (+ 2632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:32.545641 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.547789 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.547795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:32.547984 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.548014 (+ 30us) client_negotiation.cc:769] Sending connection context
0618 22:43:32.548091 (+ 77us) client_negotiation.cc:240] Negotiation successful
0618 22:43:32.548201 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13859,"mutex_wait_us":567,"spinlock_wait_cycles":8448,"thread_start_us":12390,"threads_started":1}
I20260618 22:43:32.549196 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.525613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.5.213.194:32899)
0618 22:43:32.537300 (+ 11687us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:32.537312 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:32.539276 (+ 1964us) server_negotiation.cc:407] Connection header received
0618 22:43:32.539388 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:32.539396 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:32.539823 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:32.540031 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:32.541239 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.542829 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.545829 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.547309 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.547435 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.547664 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:32.548800 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0618 22:43:32.548951 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11504,"thread_start_us":11365,"threads_started":1}
I20260618 22:43:32.548601 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.524746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:45489 (local address 127.0.0.1:55206)
0618 22:43:32.537902 (+ 13156us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:32.537955 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:32.538184 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:32.540291 (+ 2107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:32.540298 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:32.540346 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:32.540997 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:32.541015 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.542990 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.542995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:32.545625 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:32.545641 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.547443 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.547449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:32.547683 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.547738 (+ 55us) client_negotiation.cc:769] Sending connection context
0618 22:43:32.547892 (+ 154us) client_negotiation.cc:240] Negotiation successful
0618 22:43:32.548098 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12844,"mutex_wait_us":103,"spinlock_wait_cycles":8704,"thread_start_us":14013,"threads_started":1}
I20260618 22:43:32.552963 25975 tablet_service.cc:1467] Processing CreateTablet for tablet 21714465f67840878ed46587aff3a0fd (DEFAULT_TABLE table=test-table [id=2bb87f16ec5341059ac4c3786027c036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:43:32.552963 26106 tablet_service.cc:1467] Processing CreateTablet for tablet 21714465f67840878ed46587aff3a0fd (DEFAULT_TABLE table=test-table [id=2bb87f16ec5341059ac4c3786027c036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:43:32.554338 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.540241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:39187 (local address 127.0.0.1:36278)
0618 22:43:32.543937 (+ 3696us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:32.543953 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:32.544067 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:32.544592 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:32.544597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:32.544620 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:32.544981 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:32.544990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.546561 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.546566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:32.548534 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:32.548549 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.550368 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.550374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:32.550537 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.553655 (+ 3118us) client_negotiation.cc:769] Sending connection context
0618 22:43:32.553780 (+ 125us) client_negotiation.cc:240] Negotiation successful
0618 22:43:32.553938 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3593,"mutex_wait_us":1,"thread_start_us":3435,"threads_started":1}
I20260618 22:43:32.554406 25975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21714465f67840878ed46587aff3a0fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:32.554409 26106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21714465f67840878ed46587aff3a0fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:32.554401 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.538764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36278 (local address 127.5.213.195:39187)
0618 22:43:32.543448 (+ 4684us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:32.543456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:32.544049 (+ 593us) server_negotiation.cc:407] Connection header received
0618 22:43:32.544217 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:32.544225 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:32.544422 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:32.544584 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:32.545131 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.546402 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.548829 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.549894 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.550023 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.550237 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:32.553871 (+ 3634us) server_negotiation.cc:299] Negotiation successful
0618 22:43:32.554071 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4514,"thread_start_us":4365,"threads_started":1}
I20260618 22:43:32.559047 26236 tablet_service.cc:1467] Processing CreateTablet for tablet 21714465f67840878ed46587aff3a0fd (DEFAULT_TABLE table=test-table [id=2bb87f16ec5341059ac4c3786027c036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:43:32.560623 26236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21714465f67840878ed46587aff3a0fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:32.566222 26319 tablet_bootstrap.cc:492] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239: Bootstrap starting.
I20260618 22:43:32.569352 26319 tablet_bootstrap.cc:654] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:32.570343 26319 log.cc:824] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:32.571467 26320 tablet_bootstrap.cc:492] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e: Bootstrap starting.
I20260618 22:43:32.575346 26322 tablet_bootstrap.cc:492] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b: Bootstrap starting.
I20260618 22:43:32.575708 26320 tablet_bootstrap.cc:654] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:32.575771 26319 tablet_bootstrap.cc:492] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239: No bootstrap required, opened a new log
I20260618 22:43:32.576063 26319 ts_tablet_manager.cc:1397] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260618 22:43:32.577041 26320 log.cc:824] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:32.578436 26322 tablet_bootstrap.cc:654] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:32.579385 26322 log.cc:824] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:32.580183 26320 tablet_bootstrap.cc:492] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e: No bootstrap required, opened a new log
I20260618 22:43:32.580461 26320 ts_tablet_manager.cc:1397] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260618 22:43:32.582186 26322 tablet_bootstrap.cc:492] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b: No bootstrap required, opened a new log
I20260618 22:43:32.582412 26322 ts_tablet_manager.cc:1397] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260618 22:43:32.586030 26319 raft_consensus.cc:348] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678989c097da44258201f7814841ea7e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45489 } } peers { permanent_uuid: "b0cfcaa131a240d1b162ddbac4a89239" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32899 } } peers { permanent_uuid: "56266490c8834ab4b020d5b29d95f95b" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 39187 } }
I20260618 22:43:32.586498 26319 raft_consensus.cc:374] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:32.586670 26319 raft_consensus.cc:729] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0cfcaa131a240d1b162ddbac4a89239, State: Initialized, Role: FOLLOWER
I20260618 22:43:32.587335 26319 consensus_queue.cc:260] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678989c097da44258201f7814841ea7e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45489 } } peers { permanent_uuid: "b0cfcaa131a240d1b162ddbac4a89239" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32899 } } peers { permanent_uuid: "56266490c8834ab4b020d5b29d95f95b" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 39187 } }
I20260618 22:43:32.590245 26322 raft_consensus.cc:348] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678989c097da44258201f7814841ea7e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45489 } } peers { permanent_uuid: "b0cfcaa131a240d1b162ddbac4a89239" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32899 } } peers { permanent_uuid: "56266490c8834ab4b020d5b29d95f95b" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 39187 } }
I20260618 22:43:32.591466 26322 raft_consensus.cc:374] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:32.591624 26322 raft_consensus.cc:729] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56266490c8834ab4b020d5b29d95f95b, State: Initialized, Role: FOLLOWER
I20260618 22:43:32.591318 26320 raft_consensus.cc:348] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678989c097da44258201f7814841ea7e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45489 } } peers { permanent_uuid: "b0cfcaa131a240d1b162ddbac4a89239" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32899 } } peers { permanent_uuid: "56266490c8834ab4b020d5b29d95f95b" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 39187 } }
I20260618 22:43:32.591805 26320 raft_consensus.cc:374] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:32.591909 26320 raft_consensus.cc:729] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678989c097da44258201f7814841ea7e, State: Initialized, Role: FOLLOWER
I20260618 22:43:32.592440 26322 consensus_queue.cc:260] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678989c097da44258201f7814841ea7e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45489 } } peers { permanent_uuid: "b0cfcaa131a240d1b162ddbac4a89239" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32899 } } peers { permanent_uuid: "56266490c8834ab4b020d5b29d95f95b" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 39187 } }
I20260618 22:43:32.592644 26320 consensus_queue.cc:260] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678989c097da44258201f7814841ea7e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45489 } } peers { permanent_uuid: "b0cfcaa131a240d1b162ddbac4a89239" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32899 } } peers { permanent_uuid: "56266490c8834ab4b020d5b29d95f95b" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 39187 } }
I20260618 22:43:32.594736 26171 heartbeater.cc:503] Master 127.5.213.254:37715 was elected leader, sending a full tablet report...
I20260618 22:43:32.598255 26320 ts_tablet_manager.cc:1428] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e: Time spent starting tablet: real 0.018s user 0.010s sys 0.006s
I20260618 22:43:32.598655 26040 heartbeater.cc:503] Master 127.5.213.254:37715 was elected leader, sending a full tablet report...
I20260618 22:43:32.600499 26319 ts_tablet_manager.cc:1428] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239: Time spent starting tablet: real 0.024s user 0.010s sys 0.008s
I20260618 22:43:32.600744 26322 ts_tablet_manager.cc:1428] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b: Time spent starting tablet: real 0.018s user 0.006s sys 0.013s
I20260618 22:43:32.601099 26301 heartbeater.cc:503] Master 127.5.213.254:37715 was elected leader, sending a full tablet report...
I20260618 22:43:32.675616 26325 raft_consensus.cc:482] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:43:32.675916 26325 raft_consensus.cc:504] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678989c097da44258201f7814841ea7e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45489 } } peers { permanent_uuid: "b0cfcaa131a240d1b162ddbac4a89239" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32899 } } peers { permanent_uuid: "56266490c8834ab4b020d5b29d95f95b" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 39187 } }
I20260618 22:43:32.677742 26325 leader_election.cc:288] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 678989c097da44258201f7814841ea7e (127.5.213.193:45489), 56266490c8834ab4b020d5b29d95f95b (127.5.213.195:39187)
I20260618 22:43:32.679767 26327 raft_consensus.cc:482] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:43:32.680119 26327 raft_consensus.cc:504] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678989c097da44258201f7814841ea7e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45489 } } peers { permanent_uuid: "b0cfcaa131a240d1b162ddbac4a89239" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32899 } } peers { permanent_uuid: "56266490c8834ab4b020d5b29d95f95b" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 39187 } }
I20260618 22:43:32.682068 26327 leader_election.cc:288] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0cfcaa131a240d1b162ddbac4a89239 (127.5.213.194:32899), 56266490c8834ab4b020d5b29d95f95b (127.5.213.195:39187)
I20260618 22:43:32.686288 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.678479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:60673 (local address 127.5.213.195:39187)
0618 22:43:32.678693 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:32.678699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:32.679180 (+ 481us) server_negotiation.cc:407] Connection header received
0618 22:43:32.679369 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:32.679373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:32.679448 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:32.679535 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:32.680156 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.681219 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.683956 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.685156 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.685277 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.685350 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:32.685888 (+ 538us) server_negotiation.cc:299] Negotiation successful
0618 22:43:32.686022 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260618 22:43:32.686435 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.678269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.194:48319 (local address 127.5.213.193:45489)
0618 22:43:32.678479 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:32.678486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:32.679290 (+ 804us) server_negotiation.cc:407] Connection header received
0618 22:43:32.679442 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:32.679446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:32.679532 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:32.679615 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:32.680367 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.681402 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.683995 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.685278 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.685396 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.685463 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:32.685976 (+ 513us) server_negotiation.cc:299] Negotiation successful
0618 22:43:32.686119 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260618 22:43:32.687111 26328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.678163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.193:45489 (local address 127.5.213.194:48319)
0618 22:43:32.679164 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:32.679182 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:32.679310 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:32.679720 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:32.679725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:32.679765 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:32.680201 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:32.680214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.681530 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.681535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:32.683753 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:32.683769 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.685633 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.685639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:32.685806 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.685837 (+ 31us) client_negotiation.cc:769] Sending connection context
0618 22:43:32.686578 (+ 741us) client_negotiation.cc:240] Negotiation successful
0618 22:43:32.686751 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"mutex_wait_us":49,"thread_start_us":581,"threads_started":1}
I20260618 22:43:32.687511 26329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.678049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:39187 (local address 127.5.213.194:60673)
0618 22:43:32.679074 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:32.679091 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:32.679237 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:32.679561 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:32.679566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:32.679590 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:32.680013 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:32.680026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.681363 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.681368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:32.683736 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:32.683748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.685515 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.685522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:32.685707 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.685741 (+ 34us) client_negotiation.cc:769] Sending connection context
0618 22:43:32.686850 (+ 1109us) client_negotiation.cc:240] Negotiation successful
0618 22:43:32.686981 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"mutex_wait_us":36,"thread_start_us":743,"threads_started":1}
I20260618 22:43:32.688462 25995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21714465f67840878ed46587aff3a0fd" candidate_uuid: "b0cfcaa131a240d1b162ddbac4a89239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678989c097da44258201f7814841ea7e" is_pre_election: true
I20260618 22:43:32.688589 26256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21714465f67840878ed46587aff3a0fd" candidate_uuid: "b0cfcaa131a240d1b162ddbac4a89239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56266490c8834ab4b020d5b29d95f95b" is_pre_election: true
I20260618 22:43:32.688918 25995 raft_consensus.cc:2454] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0cfcaa131a240d1b162ddbac4a89239 in term 0.
I20260618 22:43:32.689200 26256 raft_consensus.cc:2454] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0cfcaa131a240d1b162ddbac4a89239 in term 0.
I20260618 22:43:32.689581 26060 leader_election.cc:302] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678989c097da44258201f7814841ea7e, b0cfcaa131a240d1b162ddbac4a89239; no voters:
I20260618 22:43:32.690042 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.682447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:57537 (local address 127.5.213.194:32899)
0618 22:43:32.682700 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:32.682705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:32.683193 (+ 488us) server_negotiation.cc:407] Connection header received
0618 22:43:32.683442 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:32.683447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:32.683527 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:32.683583 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:32.684396 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.685409 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.687902 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.688958 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.689082 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.689141 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:32.689706 (+ 565us) server_negotiation.cc:299] Negotiation successful
0618 22:43:32.689824 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260618 22:43:32.690295 26325 raft_consensus.cc:2781] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:43:32.690438 26325 raft_consensus.cc:482] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:43:32.690536 26325 raft_consensus.cc:3037] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:32.690621 26330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.682242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.194:32899 (local address 127.5.213.193:57537)
0618 22:43:32.683102 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:32.683121 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:32.683299 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:32.683765 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:32.683770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:32.683797 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:32.684217 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:32.684243 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.685569 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.685574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:32.687685 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:32.687700 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.689263 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.689269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:32.689497 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.689537 (+ 40us) client_negotiation.cc:769] Sending connection context
0618 22:43:32.690176 (+ 639us) client_negotiation.cc:240] Negotiation successful
0618 22:43:32.690304 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":4,"thread_start_us":587,"threads_started":1}
I20260618 22:43:32.691512 26126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21714465f67840878ed46587aff3a0fd" candidate_uuid: "678989c097da44258201f7814841ea7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0cfcaa131a240d1b162ddbac4a89239" is_pre_election: true
I20260618 22:43:32.693051 26325 raft_consensus.cc:504] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678989c097da44258201f7814841ea7e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45489 } } peers { permanent_uuid: "b0cfcaa131a240d1b162ddbac4a89239" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32899 } } peers { permanent_uuid: "56266490c8834ab4b020d5b29d95f95b" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 39187 } }
I20260618 22:43:32.693532 26126 raft_consensus.cc:2379] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 678989c097da44258201f7814841ea7e in current term 1: Already voted for candidate b0cfcaa131a240d1b162ddbac4a89239 in this term.
I20260618 22:43:32.694326 25995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21714465f67840878ed46587aff3a0fd" candidate_uuid: "b0cfcaa131a240d1b162ddbac4a89239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678989c097da44258201f7814841ea7e"
I20260618 22:43:32.694373 26256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21714465f67840878ed46587aff3a0fd" candidate_uuid: "b0cfcaa131a240d1b162ddbac4a89239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56266490c8834ab4b020d5b29d95f95b"
I20260618 22:43:32.694548 25995 raft_consensus.cc:3037] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:32.694591 26256 raft_consensus.cc:3037] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:32.695297 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.683599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.213.193:56723 (local address 127.5.213.195:39187)
0618 22:43:32.686509 (+ 2910us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:32.686514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:32.687999 (+ 1485us) server_negotiation.cc:407] Connection header received
0618 22:43:32.688899 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:32.688904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:32.688998 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:32.689054 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:43:32.690163 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.691179 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.692977 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.694114 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.694274 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.694338 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:32.694878 (+ 540us) server_negotiation.cc:299] Negotiation successful
0618 22:43:32.695008 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2814,"thread_start_us":5838,"threads_started":1}
I20260618 22:43:32.695943 26331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.683204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.195:39187 (local address 127.5.213.193:56723)
0618 22:43:32.687914 (+ 4710us) negotiation.cc:107] Waiting for socket to connect
0618 22:43:32.687930 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:43:32.688090 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:43:32.689306 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:43:32.689310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:43:32.689336 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:43:32.690005 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:32.690017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.691321 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.691326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:32.692801 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:43:32.692815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.694455 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.694502 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:43:32.694693 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.694726 (+ 33us) client_negotiation.cc:769] Sending connection context
0618 22:43:32.695455 (+ 729us) client_negotiation.cc:240] Negotiation successful
0618 22:43:32.695612 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4587,"mutex_wait_us":96,"thread_start_us":4351,"threads_started":1}
I20260618 22:43:32.696651 26255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21714465f67840878ed46587aff3a0fd" candidate_uuid: "678989c097da44258201f7814841ea7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56266490c8834ab4b020d5b29d95f95b" is_pre_election: true
I20260618 22:43:32.697418 26256 raft_consensus.cc:2454] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0cfcaa131a240d1b162ddbac4a89239 in term 1.
I20260618 22:43:32.697880 26062 leader_election.cc:302] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56266490c8834ab4b020d5b29d95f95b, b0cfcaa131a240d1b162ddbac4a89239; no voters:
I20260618 22:43:32.697880 25931 leader_election.cc:302] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 678989c097da44258201f7814841ea7e; no voters: 56266490c8834ab4b020d5b29d95f95b, b0cfcaa131a240d1b162ddbac4a89239
I20260618 22:43:32.698211 26325 leader_election.cc:288] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [CANDIDATE]: Term 1 election: Requested vote from peers 678989c097da44258201f7814841ea7e (127.5.213.193:45489), 56266490c8834ab4b020d5b29d95f95b (127.5.213.195:39187)
I20260618 22:43:32.698382 26325 raft_consensus.cc:2781] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:32.698508 26325 raft_consensus.cc:686] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [term 1 LEADER]: Becoming Leader. State: Replica: b0cfcaa131a240d1b162ddbac4a89239, State: Running, Role: LEADER
I20260618 22:43:32.697531 25995 raft_consensus.cc:2454] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0cfcaa131a240d1b162ddbac4a89239 in term 1.
I20260618 22:43:32.698797 26327 raft_consensus.cc:2726] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260618 22:43:32.698994 26325 consensus_queue.cc:237] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678989c097da44258201f7814841ea7e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45489 } } peers { permanent_uuid: "b0cfcaa131a240d1b162ddbac4a89239" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32899 } } peers { permanent_uuid: "56266490c8834ab4b020d5b29d95f95b" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 39187 } }
I20260618 22:43:32.703285 25853 catalog_manager.cc:5495] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0cfcaa131a240d1b162ddbac4a89239 (127.5.213.194). New cstate: current_term: 1 leader_uuid: "b0cfcaa131a240d1b162ddbac4a89239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678989c097da44258201f7814841ea7e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0cfcaa131a240d1b162ddbac4a89239" member_type: VOTER last_known_addr { host: "127.5.213.194" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56266490c8834ab4b020d5b29d95f95b" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 39187 } health_report { overall_health: UNKNOWN } } }
W20260618 22:43:32.723764 26302 tablet.cc:2369] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:43:32.747167 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.741812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41044 (local address 127.5.213.194:32899)
0618 22:43:32.742012 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:32.742016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:32.742030 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:43:32.742132 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:32.742136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:32.742232 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:32.742392 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:43:32.742838 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.743647 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.745108 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.745513 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.745617 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.746105 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:43:32.746598 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:43:32.746769 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:32.746816 (+ 47us) server_negotiation.cc:299] Negotiation successful
0618 22:43:32.746893 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260618 22:43:32.754863 26256 raft_consensus.cc:1261] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b [term 1 FOLLOWER]: Refusing update from remote peer b0cfcaa131a240d1b162ddbac4a89239: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:43:32.754973 25995 raft_consensus.cc:1261] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e [term 1 FOLLOWER]: Refusing update from remote peer b0cfcaa131a240d1b162ddbac4a89239: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:43:32.755788 26333 consensus_queue.cc:1035] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "678989c097da44258201f7814841ea7e" member_type: VOTER last_known_addr { host: "127.5.213.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:43:32.756119 26325 consensus_queue.cc:1035] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56266490c8834ab4b020d5b29d95f95b" member_type: VOTER last_known_addr { host: "127.5.213.195" port: 39187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:43:32.769215 26337 mvcc.cc:205] Tried to move back new op lower bound from 7298345421834731520 to 7298345421618982912. Current Snapshot: MvccSnapshot[applied={T|T < 7298345421834731520}]
I20260618 22:43:32.780829 25854 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:43:32.781127 25854 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:43:32.784883 25854 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74780dcb3e614f66ab5f3e8be3f65b6f: Sending DeleteTablet for 3 replicas of tablet 21714465f67840878ed46587aff3a0fd
I20260618 22:43:32.786105 25975 tablet_service.cc:1514] Processing DeleteTablet for tablet 21714465f67840878ed46587aff3a0fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:43:32 UTC) from {username='slave'} at 127.0.0.1:55206
I20260618 22:43:32.786466 26106 tablet_service.cc:1514] Processing DeleteTablet for tablet 21714465f67840878ed46587aff3a0fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:43:32 UTC) from {username='slave'} at 127.0.0.1:41042
I20260618 22:43:32.786908 26236 tablet_service.cc:1514] Processing DeleteTablet for tablet 21714465f67840878ed46587aff3a0fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:43:32 UTC) from {username='slave'} at 127.0.0.1:36278
I20260618 22:43:32.788312 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 25913
I20260618 22:43:32.798807 26347 tablet_replica.cc:330] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239: stopping tablet replica
I20260618 22:43:32.799294 26347 raft_consensus.cc:2229] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:43:32.800007 26347 raft_consensus.cc:2258] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:43:32.800788 26346 tablet_replica.cc:330] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e: stopping tablet replica
I20260618 22:43:32.801774 26346 raft_consensus.cc:2229] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:43:32.802300 26346 raft_consensus.cc:2258] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:43:32.802773 26347 ts_tablet_manager.cc:1905] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:43:32.805138 26346 ts_tablet_manager.cc:1905] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:43:32.810516 26347 ts_tablet_manager.cc:1918] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:43:32.810818 26347 log.cc:1196] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-1/wal/wals/21714465f67840878ed46587aff3a0fd
I20260618 22:43:32.811446 26347 ts_tablet_manager.cc:1939] T 21714465f67840878ed46587aff3a0fd P b0cfcaa131a240d1b162ddbac4a89239: Deleting consensus metadata
I20260618 22:43:32.813548 25841 catalog_manager.cc:4841] TS b0cfcaa131a240d1b162ddbac4a89239 (127.5.213.194:32899): tablet 21714465f67840878ed46587aff3a0fd (table test-table [id=2bb87f16ec5341059ac4c3786027c036]) successfully deleted
I20260618 22:43:32.814467 26348 tablet_replica.cc:330] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b: stopping tablet replica
I20260618 22:43:32.814944 26348 raft_consensus.cc:2229] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:43:32.815253 26346 ts_tablet_manager.cc:1918] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:43:32.815400 26346 log.cc:1196] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-0/wal/wals/21714465f67840878ed46587aff3a0fd
I20260618 22:43:32.815480 26348 raft_consensus.cc:2258] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:43:32.815902 26346 ts_tablet_manager.cc:1939] T 21714465f67840878ed46587aff3a0fd P 678989c097da44258201f7814841ea7e: Deleting consensus metadata
I20260618 22:43:32.817735 25839 catalog_manager.cc:4841] TS 678989c097da44258201f7814841ea7e (127.5.213.193:45489): tablet 21714465f67840878ed46587aff3a0fd (table test-table [id=2bb87f16ec5341059ac4c3786027c036]) successfully deleted
I20260618 22:43:32.819371 26348 ts_tablet_manager.cc:1905] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:43:32.821336 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.789952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.5.213.193:45489)
0618 22:43:32.790151 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:32.790156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:32.803129 (+ 12973us) server_negotiation.cc:407] Connection header received
0618 22:43:32.807037 (+ 3908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:32.807044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:32.807154 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:32.807252 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:32.813998 (+ 6746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.815194 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.817600 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.818181 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.818278 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.819748 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:32.819759 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:32.819777 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:32.820505 (+ 728us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:32.820564 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:32.820588 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:32.820705 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:32.820867 (+ 162us) server_negotiation.cc:299] Negotiation successful
0618 22:43:32.820980 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260618 22:43:32.821970 26035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:32.825513 26348 ts_tablet_manager.cc:1918] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:43:32.825651 26348 log.cc:1196] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b: Deleting WAL directory at /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781822307836422-5975-0/minicluster-data/ts-2/wal/wals/21714465f67840878ed46587aff3a0fd
I20260618 22:43:32.826038 26348 ts_tablet_manager.cc:1939] T 21714465f67840878ed46587aff3a0fd P 56266490c8834ab4b020d5b29d95f95b: Deleting consensus metadata
I20260618 22:43:32.827493 25841 catalog_manager.cc:4841] TS 56266490c8834ab4b020d5b29d95f95b (127.5.213.195:39187): tablet 21714465f67840878ed46587aff3a0fd (table test-table [id=2bb87f16ec5341059ac4c3786027c036]) successfully deleted
I20260618 22:43:32.939054 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 25913
I20260618 22:43:32.953648 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 26044
I20260618 22:43:32.960477 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:32.954452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41054 (local address 127.5.213.194:32899)
0618 22:43:32.954659 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:32.954665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:32.954685 (+ 20us) server_negotiation.cc:407] Connection header received
0618 22:43:32.954805 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:32.954810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:32.954918 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:32.955017 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:32.955422 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.956164 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.957427 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:32.957858 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:32.957975 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:32.959081 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:32.959091 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:32.959106 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:32.959745 (+ 639us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:32.959803 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:32.959824 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:32.959924 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:32.960057 (+ 133us) server_negotiation.cc:299] Negotiation successful
0618 22:43:32.960162 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:43:32.961221 26166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:33.073436 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 26044
I20260618 22:43:33.088232 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 26174
I20260618 22:43:33.094605 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:33.088964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36282 (local address 127.5.213.195:39187)
0618 22:43:33.089114 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:33.089118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:33.089156 (+ 38us) server_negotiation.cc:407] Connection header received
0618 22:43:33.089341 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:33.089345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:33.089433 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:33.089518 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:33.089914 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:33.090588 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:33.091880 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:33.092271 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:33.092362 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:33.093298 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:33.093307 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:33.093321 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:33.093936 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:33.093987 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:33.094008 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:33.094091 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:33.094221 (+ 130us) server_negotiation.cc:299] Negotiation successful
0618 22:43:33.094311 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:43:33.095323 26296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:33.182725 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 26174
I20260618 22:43:33.195643 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 25823
I20260618 22:43:33.196730 25884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:33.303800 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 25823
2026-06-18T22:43:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2744 ms)
[----------] 6 tests from EncryptionPolicyTest (20613 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:43:33 dist-test-slave-lnj3 krb5kdc[26356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:43:33 dist-test-slave-lnj3 krb5kdc[26356](info): set up 2 sockets
Jun 18 22:43:33 dist-test-slave-lnj3 krb5kdc[26356](info): commencing operation
krb5kdc: starting...
W20260618 22:43:34.327026 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:43:34 dist-test-slave-lnj3 krb5kdc[26356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:43:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:43:34.566241 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43687
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:34.889878 26369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:34.890211 26369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:34.890350 26369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:34.897814 26369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:43:34.897960 26369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:34.898043 26369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:34.898124 26369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:43:34.898181 26369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:43:34.908281 26369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43687
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:34.909586 26369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:34.910949 26369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:34.922487 26374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:34.922519 26375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:34.922842 26377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:34.923527 26369 server_base.cc:1034] running on GCE node
I20260618 22:43:34.924417 26369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:34.925943 26369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:34.927184 26369 hybrid_clock.cc:648] HybridClock initialized: now 1781822614927116 us; error 56 us; skew 500 ppm
I20260618 22:43:34.927552 26369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:34 dist-test-slave-lnj3 krb5kdc[26356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:34.930729 26369 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:43:34.932210 26369 webserver.cc:460] Webserver started at http://127.0.0.1:46657/ using document root <none> and password file <none>
I20260618 22:43:34.932910 26369 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:34.933022 26369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:34.933305 26369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:34.936415 26369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "1fa884be703641a7884c02c96c347215"
format_stamp: "Formatted at 2026-06-18 22:43:34 on dist-test-slave-lnj3"
server_key: "039a20027c2134ed03c7e32e3f15db3e"
server_key_iv: "1d2fb04a67f766cfc98cf4bdeecd3f75"
server_key_version: "encryptionkey@0"
I20260618 22:43:34.937217 26369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "1fa884be703641a7884c02c96c347215"
format_stamp: "Formatted at 2026-06-18 22:43:34 on dist-test-slave-lnj3"
server_key: "039a20027c2134ed03c7e32e3f15db3e"
server_key_iv: "1d2fb04a67f766cfc98cf4bdeecd3f75"
server_key_version: "encryptionkey@0"
I20260618 22:43:34.942581 26369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260618 22:43:34.946440 26384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:34.948035 26369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:43:34.948230 26369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "1fa884be703641a7884c02c96c347215"
format_stamp: "Formatted at 2026-06-18 22:43:34 on dist-test-slave-lnj3"
server_key: "039a20027c2134ed03c7e32e3f15db3e"
server_key_iv: "1d2fb04a67f766cfc98cf4bdeecd3f75"
server_key_version: "encryptionkey@0"
I20260618 22:43:34.948428 26369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:34.963717 26369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:34.967792 26369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:34.968047 26369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:34.985358 26369 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43687
I20260618 22:43:34.985402 26435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43687 every 8 connection(s)
I20260618 22:43:34.986763 26369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:43:34.989569 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 26369
I20260618 22:43:34.989758 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:43:34.990375 5975 external_mini_cluster.cc:1428] Setting key 29b00a28560b1ec729edc904153ff114
I20260618 22:43:34.990513 26436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:35.004568 26436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215: Bootstrap starting.
Jun 18 22:43:34 dist-test-slave-lnj3 krb5kdc[26356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260618 22:43:35.008194 26436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:35.009227 26436 log.cc:824] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:35.012297 26436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215: No bootstrap required, opened a new log
I20260618 22:43:35.013650 26439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:34.992319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60995 (local address 127.0.0.1:43687)
0618 22:43:34.993065 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:34.993077 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:34.993120 (+ 43us) server_negotiation.cc:407] Connection header received
0618 22:43:34.995045 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:34.995067 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:34.995500 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:34.995831 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:34.997558 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:34.998917 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:35.000691 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:35.001442 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:35.001619 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:35.006546 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:35.006555 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:35.006572 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:35.006615 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:35.009669 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:35.010467 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:35.010473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:35.010480 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:35.010659 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:35.011108 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:35.011111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:35.011113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:35.011610 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:35.011831 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:35.012319 (+ 488us) server_negotiation.cc:299] Negotiation successful
0618 22:43:35.012648 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":327,"threads_started":1}
I20260618 22:43:35.017537 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:43:35.021279 26436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa884be703641a7884c02c96c347215" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260618 22:43:35.021783 26436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:35.021885 26436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa884be703641a7884c02c96c347215, State: Initialized, Role: FOLLOWER
I20260618 22:43:35.022543 26436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa884be703641a7884c02c96c347215" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260618 22:43:35.022831 26436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:43:35.023016 26436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:43:35.023232 26436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:35.026301 26436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa884be703641a7884c02c96c347215" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260618 22:43:35.026878 26436 leader_election.cc:302] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fa884be703641a7884c02c96c347215; no voters:
I20260618 22:43:35.027522 26436 leader_election.cc:288] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:43:35.027613 26446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:35.028764 26446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215 [term 1 LEADER]: Becoming Leader. State: Replica: 1fa884be703641a7884c02c96c347215, State: Running, Role: LEADER
I20260618 22:43:35.029357 26446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa884be703641a7884c02c96c347215" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260618 22:43:35.029752 26436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:43:35.032246 26448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fa884be703641a7884c02c96c347215. Latest consensus state: current_term: 1 leader_uuid: "1fa884be703641a7884c02c96c347215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa884be703641a7884c02c96c347215" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } } }
I20260618 22:43:35.032260 26447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fa884be703641a7884c02c96c347215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa884be703641a7884c02c96c347215" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } } }
I20260618 22:43:35.032716 26448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:35.032716 26447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:35.033562 26451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:43:35.040756 26439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:35.023074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47737 (local address 127.0.0.1:43687)
0618 22:43:35.023259 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:35.023266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:35.023829 (+ 563us) server_negotiation.cc:407] Connection header received
0618 22:43:35.023983 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:35.023987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:35.024053 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:35.024137 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:35.025869 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:35.026744 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:35.028462 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:35.029147 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:35.029261 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:35.032713 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:35.032722 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:35.032728 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:35.032773 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:35.036714 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:35.037820 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:35.037826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:35.037830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:35.038012 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:35.038578 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:35.038584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:35.038588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:35.039137 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:35.039421 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:35.040134 (+ 713us) server_negotiation.cc:299] Negotiation successful
0618 22:43:35.040346 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260618 22:43:35.047601 26451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:43:35.056602 26451 catalog_manager.cc:1347] Generated new cluster ID: a7d53eeeee6a469fbdee764d9b47bbda
I20260618 22:43:35.056717 26451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:43:35.066982 26451 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:43:35.068337 26451 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:43:35.077641 26451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fa884be703641a7884c02c96c347215: Generated new TSK 0
I20260618 22:43:35.078756 26451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:43:35.091492 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 26369
I20260618 22:43:35.093101 26431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:35.182399 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 26369
W20260618 22:43:35.196141 26354 connection.cc:535] client connection to 127.0.0.1:43687 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43687): Connection reset by peer (error 104)
2026-06-18T22:43:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1874 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:43:35 dist-test-slave-lnj3 krb5kdc[26468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:43:35 dist-test-slave-lnj3 krb5kdc[26468](info): set up 2 sockets
Jun 18 22:43:35 dist-test-slave-lnj3 krb5kdc[26468](info): commencing operation
krb5kdc: starting...
W20260618 22:43:36.206779 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:43:36 dist-test-slave-lnj3 krb5kdc[26468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:43:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:43:36.446345 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33119
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:36.774755 26481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:36.775053 26481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:36.775144 26481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:36.782361 26481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:43:36.782480 26481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:36.782524 26481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:36.782585 26481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:43:36.782645 26481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:43:36.792621 26481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33119
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:36.793846 26481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:36.795145 26481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:36.806155 26486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:36.806751 26489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:36.806809 26487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:36.807272 26481 server_base.cc:1034] running on GCE node
I20260618 22:43:36.808022 26481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:36.809406 26481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:36.810597 26481 hybrid_clock.cc:648] HybridClock initialized: now 1781822616810532 us; error 55 us; skew 500 ppm
I20260618 22:43:36.810951 26481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:36 dist-test-slave-lnj3 krb5kdc[26468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:36.813894 26481 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:43:36.815374 26481 webserver.cc:460] Webserver started at http://127.0.0.1:39261/ using document root <none> and password file <none>
I20260618 22:43:36.816041 26481 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:36.816156 26481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:36.816450 26481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:36.819473 26481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "f4ba882c51674d1b99def432cff0119e"
format_stamp: "Formatted at 2026-06-18 22:43:36 on dist-test-slave-lnj3"
server_key: "f401bd27c8c42c494c1d36b834ca31bb"
server_key_iv: "62ffc754fafd28e87c314116bfc50c22"
server_key_version: "encryptionkey@0"
I20260618 22:43:36.820232 26481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "f4ba882c51674d1b99def432cff0119e"
format_stamp: "Formatted at 2026-06-18 22:43:36 on dist-test-slave-lnj3"
server_key: "f401bd27c8c42c494c1d36b834ca31bb"
server_key_iv: "62ffc754fafd28e87c314116bfc50c22"
server_key_version: "encryptionkey@0"
I20260618 22:43:36.825470 26481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260618 22:43:36.829473 26496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:36.830904 26481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260618 22:43:36.831060 26481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "f4ba882c51674d1b99def432cff0119e"
format_stamp: "Formatted at 2026-06-18 22:43:36 on dist-test-slave-lnj3"
server_key: "f401bd27c8c42c494c1d36b834ca31bb"
server_key_iv: "62ffc754fafd28e87c314116bfc50c22"
server_key_version: "encryptionkey@0"
I20260618 22:43:36.831246 26481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:36.855182 26481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:36.859367 26481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:36.859632 26481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:36.876904 26481 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33119
I20260618 22:43:36.876933 26547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33119 every 8 connection(s)
I20260618 22:43:36.878268 26481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:43:36.879817 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 26481
I20260618 22:43:36.879988 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:43:36.880528 5975 external_mini_cluster.cc:1428] Setting key de2b970de2ee066366371c921ee01b91
I20260618 22:43:36.882804 26548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:36.893715 26551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:36.882351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37589 (local address 127.0.0.1:33119)
0618 22:43:36.883141 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:36.883154 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:36.883197 (+ 43us) server_negotiation.cc:407] Connection header received
0618 22:43:36.884801 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:36.884822 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:36.885211 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:36.885532 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:36.886068 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:36.887441 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:36.889261 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:36.889957 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:36.890083 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:36.891389 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:36.891396 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:36.891410 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:36.892001 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:36.892083 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:36.892110 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:36.892229 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:36.892514 (+ 285us) server_negotiation.cc:299] Negotiation successful
0618 22:43:36.892748 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":357,"threads_started":1}
I20260618 22:43:36.894860 26548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e: Bootstrap starting.
I20260618 22:43:36.897432 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:43:36.898048 26548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:36.899027 26548 log.cc:824] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:36.902058 26548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e: No bootstrap required, opened a new log
I20260618 22:43:36.907450 26551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:36.901694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36839 (local address 127.0.0.1:33119)
0618 22:43:36.901842 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:36.901848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:36.902238 (+ 390us) server_negotiation.cc:407] Connection header received
0618 22:43:36.902421 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:36.902425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:36.902488 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:36.902566 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:36.903093 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:36.903872 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:36.905171 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:36.905591 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:36.905718 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:36.906794 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:36.906799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:36.906802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:36.906818 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:36.906857 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:36.906875 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:36.906964 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:36.907157 (+ 193us) server_negotiation.cc:299] Negotiation successful
0618 22:43:36.907215 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:43:36.909703 26548 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ba882c51674d1b99def432cff0119e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33119 } }
I20260618 22:43:36.910036 26548 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:36.910109 26548 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4ba882c51674d1b99def432cff0119e, State: Initialized, Role: FOLLOWER
I20260618 22:43:36.910681 26548 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ba882c51674d1b99def432cff0119e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33119 } }
I20260618 22:43:36.910884 26548 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:43:36.910979 26548 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:43:36.911103 26548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:36.913121 26548 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ba882c51674d1b99def432cff0119e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33119 } }
I20260618 22:43:36.913558 26548 leader_election.cc:302] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4ba882c51674d1b99def432cff0119e; no voters:
I20260618 22:43:36.914063 26548 leader_election.cc:288] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:43:36.914198 26558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:36.915284 26558 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e [term 1 LEADER]: Becoming Leader. State: Replica: f4ba882c51674d1b99def432cff0119e, State: Running, Role: LEADER
I20260618 22:43:36.915820 26558 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ba882c51674d1b99def432cff0119e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33119 } }
I20260618 22:43:36.916070 26548 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:43:36.918562 26559 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4ba882c51674d1b99def432cff0119e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ba882c51674d1b99def432cff0119e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33119 } } }
I20260618 22:43:36.919050 26559 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:36.918655 26560 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4ba882c51674d1b99def432cff0119e. Latest consensus state: current_term: 1 leader_uuid: "f4ba882c51674d1b99def432cff0119e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ba882c51674d1b99def432cff0119e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33119 } } }
I20260618 22:43:36.919571 26560 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:36.920826 26567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:43:36.926306 26567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:43:36.935983 26567 catalog_manager.cc:1347] Generated new cluster ID: 793f51ca8b37401287ce4525a291f4e7
I20260618 22:43:36.936100 26567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:43:36.944470 26567 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:43:36.945832 26567 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:43:36.953708 26567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4ba882c51674d1b99def432cff0119e: Generated new TSK 0
I20260618 22:43:36.954743 26567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:43:36.970227 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 26481
W20260618 22:43:36.971407 26543 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37589
W20260618 22:43:36.972118 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 26481: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:43:36.972317 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 26481
W20260618 22:43:36.985965 26466 connection.cc:535] client connection to 127.0.0.1:33119 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33119): Connection reset by peer (error 104)
2026-06-18T22:43:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1789 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:43:37 dist-test-slave-lnj3 krb5kdc[26579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:43:37 dist-test-slave-lnj3 krb5kdc[26579](info): set up 2 sockets
Jun 18 22:43:37 dist-test-slave-lnj3 krb5kdc[26579](info): commencing operation
krb5kdc: starting...
W20260618 22:43:37.978843 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:43:38 dist-test-slave-lnj3 krb5kdc[26579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:43:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:43:38.216120 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44661
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:38.534056 26592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:38.534363 26592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:38.534499 26592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:38.541759 26592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:43:38.541888 26592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:38.541968 26592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:38.542045 26592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:43:38.542104 26592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:43:38.552106 26592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44661
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:38.553390 26592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:38.554682 26592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:38.566041 26597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:38.566124 26598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:38.566193 26600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:38.567229 26592 server_base.cc:1034] running on GCE node
I20260618 22:43:38.567977 26592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:38.569396 26592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:38.570591 26592 hybrid_clock.cc:648] HybridClock initialized: now 1781822618570531 us; error 57 us; skew 500 ppm
I20260618 22:43:38.570950 26592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:38 dist-test-slave-lnj3 krb5kdc[26579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:38.578540 26592 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:43:38.579962 26592 webserver.cc:460] Webserver started at http://127.0.0.1:33317/ using document root <none> and password file <none>
I20260618 22:43:38.580600 26592 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:38.580693 26592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:38.580933 26592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:38.583812 26592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "0a34d646780143a0ac57ae646514ae73"
format_stamp: "Formatted at 2026-06-18 22:43:38 on dist-test-slave-lnj3"
server_key: "e6f07fff50013f08258119f782484be5"
server_key_iv: "5d254b26ee22d26542b8ceb26c374a42"
server_key_version: "encryptionkey@0"
I20260618 22:43:38.584507 26592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "0a34d646780143a0ac57ae646514ae73"
format_stamp: "Formatted at 2026-06-18 22:43:38 on dist-test-slave-lnj3"
server_key: "e6f07fff50013f08258119f782484be5"
server_key_iv: "5d254b26ee22d26542b8ceb26c374a42"
server_key_version: "encryptionkey@0"
I20260618 22:43:38.589687 26592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260618 22:43:38.593063 26607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:38.594559 26592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:43:38.594808 26592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "0a34d646780143a0ac57ae646514ae73"
format_stamp: "Formatted at 2026-06-18 22:43:38 on dist-test-slave-lnj3"
server_key: "e6f07fff50013f08258119f782484be5"
server_key_iv: "5d254b26ee22d26542b8ceb26c374a42"
server_key_version: "encryptionkey@0"
I20260618 22:43:38.594997 26592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:38.600920 26592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:38.605258 26592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:38.605547 26592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:38.624871 26592 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44661
I20260618 22:43:38.624876 26658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44661 every 8 connection(s)
I20260618 22:43:38.626307 26592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:43:38.628448 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 26592
I20260618 22:43:38.628618 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:43:38.629022 5975 external_mini_cluster.cc:1428] Setting key ccda55d57a2b15220fab33dda86261cf
I20260618 22:43:38.630343 26659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:38.637238 26662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:38.630795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39845 (local address 127.0.0.1:44661)
0618 22:43:38.631552 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:38.631567 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:38.631625 (+ 58us) server_negotiation.cc:407] Connection header received
0618 22:43:38.633461 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:38.633488 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:38.633943 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:38.634373 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:38.635336 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:38.635344 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:38.635362 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:38.635382 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:38.635462 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:38.635488 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:38.635712 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:38.635835 (+ 123us) server_negotiation.cc:299] Negotiation successful
0618 22:43:38.636014 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":297,"threads_started":1}
I20260618 22:43:38.640691 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:43:38.644217 26659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73: Bootstrap starting.
I20260618 22:43:38.647874 26659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:38.648030 26662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:38.645807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57649 (local address 127.0.0.1:44661)
0618 22:43:38.645990 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:38.645996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:38.646231 (+ 235us) server_negotiation.cc:407] Connection header received
0618 22:43:38.646364 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:38.646369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:38.646424 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:38.646522 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:38.647408 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:38.647413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:38.647418 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:38.647435 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:38.647480 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:38.647500 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:38.647615 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:38.647696 (+ 81us) server_negotiation.cc:299] Negotiation successful
0618 22:43:38.647751 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260618 22:43:38.649055 26659 log.cc:824] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:38.651679 26659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73: No bootstrap required, opened a new log
I20260618 22:43:38.658754 26659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a34d646780143a0ac57ae646514ae73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44661 } }
I20260618 22:43:38.659111 26659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:38.659184 26659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a34d646780143a0ac57ae646514ae73, State: Initialized, Role: FOLLOWER
I20260618 22:43:38.659718 26659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a34d646780143a0ac57ae646514ae73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44661 } }
I20260618 22:43:38.659909 26659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:43:38.660073 26659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:43:38.660239 26659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:38.662302 26659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a34d646780143a0ac57ae646514ae73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44661 } }
I20260618 22:43:38.662786 26659 leader_election.cc:302] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a34d646780143a0ac57ae646514ae73; no voters:
I20260618 22:43:38.663332 26659 leader_election.cc:288] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:43:38.663457 26669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:38.664069 26669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73 [term 1 LEADER]: Becoming Leader. State: Replica: 0a34d646780143a0ac57ae646514ae73, State: Running, Role: LEADER
I20260618 22:43:38.664587 26669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a34d646780143a0ac57ae646514ae73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44661 } }
I20260618 22:43:38.665189 26659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:43:38.667182 26671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a34d646780143a0ac57ae646514ae73. Latest consensus state: current_term: 1 leader_uuid: "0a34d646780143a0ac57ae646514ae73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a34d646780143a0ac57ae646514ae73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44661 } } }
I20260618 22:43:38.667557 26671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:38.667752 26670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a34d646780143a0ac57ae646514ae73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a34d646780143a0ac57ae646514ae73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44661 } } }
I20260618 22:43:38.668592 26670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:38.669936 26676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:43:38.675019 26676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:43:38.683882 26676 catalog_manager.cc:1347] Generated new cluster ID: e13e6df5c1b84171b6c07feaf4d165d5
I20260618 22:43:38.684001 26676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:43:38.703235 26676 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:43:38.703696 26676 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:43:38.718870 26676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a34d646780143a0ac57ae646514ae73: Generated new TSK 0
I20260618 22:43:38.719713 26676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:43:38.734264 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 26592
W20260618 22:43:38.735364 26654 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39845
W20260618 22:43:38.736066 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 26592: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:43:38.736203 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 26592
2026-06-18T22:43:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1763 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:43:38 dist-test-slave-lnj3 krb5kdc[26690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:43:38 dist-test-slave-lnj3 krb5kdc[26690](info): set up 2 sockets
Jun 18 22:43:38 dist-test-slave-lnj3 krb5kdc[26690](info): commencing operation
krb5kdc: starting...
W20260618 22:43:39.746675 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:43:39 dist-test-slave-lnj3 krb5kdc[26690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:43:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:43:39.974035 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33761
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:40.286449 26703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:40.286816 26703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:40.286975 26703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:40.294235 26703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:43:40.294364 26703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:40.294443 26703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:40.294526 26703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:43:40.294585 26703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:43:40.304382 26703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:40.305677 26703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:40.306977 26703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:40.318305 26711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:40.318297 26709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:40.318791 26708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:40.319273 26703 server_base.cc:1034] running on GCE node
I20260618 22:43:40.320034 26703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:40.321408 26703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:40.322662 26703 hybrid_clock.cc:648] HybridClock initialized: now 1781822620322558 us; error 63 us; skew 500 ppm
I20260618 22:43:40.323051 26703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:40 dist-test-slave-lnj3 krb5kdc[26690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:40.326378 26703 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:43:40.327828 26703 webserver.cc:460] Webserver started at http://127.0.0.1:36105/ using document root <none> and password file <none>
I20260618 22:43:40.328491 26703 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:40.328595 26703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:40.328894 26703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:40.331910 26703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "f6d1ea418203401c8b4d66144886b070"
format_stamp: "Formatted at 2026-06-18 22:43:40 on dist-test-slave-lnj3"
server_key: "37df8e0d357090cb7f5f3549fda719a5"
server_key_iv: "63a3557bd0f21a76438a70c3ce343357"
server_key_version: "encryptionkey@0"
I20260618 22:43:40.332618 26703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "f6d1ea418203401c8b4d66144886b070"
format_stamp: "Formatted at 2026-06-18 22:43:40 on dist-test-slave-lnj3"
server_key: "37df8e0d357090cb7f5f3549fda719a5"
server_key_iv: "63a3557bd0f21a76438a70c3ce343357"
server_key_version: "encryptionkey@0"
I20260618 22:43:40.337833 26703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260618 22:43:40.341478 26718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:40.342936 26703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:43:40.343093 26703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "f6d1ea418203401c8b4d66144886b070"
format_stamp: "Formatted at 2026-06-18 22:43:40 on dist-test-slave-lnj3"
server_key: "37df8e0d357090cb7f5f3549fda719a5"
server_key_iv: "63a3557bd0f21a76438a70c3ce343357"
server_key_version: "encryptionkey@0"
I20260618 22:43:40.343289 26703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:40.371423 26703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:40.375442 26703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:40.375700 26703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:40.393118 26703 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33761
I20260618 22:43:40.393151 26769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33761 every 8 connection(s)
I20260618 22:43:40.394454 26703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:43:40.396606 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 26703
I20260618 22:43:40.396763 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:43:40.397181 5975 external_mini_cluster.cc:1428] Setting key 1df5a4271f5abae155751f63d78d338f
I20260618 22:43:40.398674 26770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 18 22:43:40 dist-test-slave-lnj3 krb5kdc[26690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260618 22:43:40.412294 26770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070: Bootstrap starting.
I20260618 22:43:40.415710 26770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:40.416961 26770 log.cc:824] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:40.419826 26770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070: No bootstrap required, opened a new log
I20260618 22:43:40.420001 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:40.398916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51471 (local address 127.0.0.1:33761)
0618 22:43:40.399692 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:40.399708 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:40.399756 (+ 48us) server_negotiation.cc:407] Connection header received
0618 22:43:40.401728 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:40.401756 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:40.402297 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:40.402812 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:40.404027 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:40.406065 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:40.407331 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:40.408371 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:40.408548 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:40.412762 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:40.412770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:40.412785 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:40.412829 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:40.415715 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:40.416569 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:40.416576 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:40.416581 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:40.416752 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:40.417198 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:40.417203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:40.417206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:40.417868 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:40.418169 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:40.418621 (+ 452us) server_negotiation.cc:299] Negotiation successful
0618 22:43:40.419025 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":300,"threads_started":1}
I20260618 22:43:40.423720 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:43:40.427917 26770 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d1ea418203401c8b4d66144886b070" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33761 } }
I20260618 22:43:40.428304 26770 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:40.428404 26770 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d1ea418203401c8b4d66144886b070, State: Initialized, Role: FOLLOWER
I20260618 22:43:40.429025 26770 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d1ea418203401c8b4d66144886b070" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33761 } }
I20260618 22:43:40.429231 26770 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:43:40.429358 26770 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:43:40.429494 26770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:40.432155 26770 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d1ea418203401c8b4d66144886b070" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33761 } }
I20260618 22:43:40.432623 26770 leader_election.cc:302] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6d1ea418203401c8b4d66144886b070; no voters:
I20260618 22:43:40.433231 26770 leader_election.cc:288] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:43:40.433388 26780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:40.434917 26780 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070 [term 1 LEADER]: Becoming Leader. State: Replica: f6d1ea418203401c8b4d66144886b070, State: Running, Role: LEADER
I20260618 22:43:40.435467 26770 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:43:40.435536 26780 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d1ea418203401c8b4d66144886b070" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33761 } }
I20260618 22:43:40.438740 26782 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6d1ea418203401c8b4d66144886b070. Latest consensus state: current_term: 1 leader_uuid: "f6d1ea418203401c8b4d66144886b070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d1ea418203401c8b4d66144886b070" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33761 } } }
I20260618 22:43:40.439527 26782 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:40.439491 26781 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6d1ea418203401c8b4d66144886b070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d1ea418203401c8b4d66144886b070" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33761 } } }
I20260618 22:43:40.439953 26781 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:40.445468 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:40.427833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36235 (local address 127.0.0.1:33761)
0618 22:43:40.428017 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:40.428023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:40.428317 (+ 294us) server_negotiation.cc:407] Connection header received
0618 22:43:40.428495 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:40.428500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:40.428574 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:40.428682 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:40.430199 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:40.431354 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:40.432781 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:40.433471 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:40.433604 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:40.436907 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:40.436913 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:40.436918 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:40.436967 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:40.441260 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:40.442214 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:40.442221 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:40.442225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:40.442402 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:40.442889 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:40.442894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:40.442898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:40.443309 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:40.443587 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:40.444784 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0618 22:43:40.445027 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 22:43:40.451457 26788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:43:40.457767 26788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:43:40.466990 26788 catalog_manager.cc:1347] Generated new cluster ID: a2431416aca3478eba50e92f243715e3
I20260618 22:43:40.467116 26788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:43:40.479362 26788 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:43:40.481307 26788 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:43:40.495445 26788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6d1ea418203401c8b4d66144886b070: Generated new TSK 0
I20260618 22:43:40.496289 26788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:43:40.504639 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 26703
I20260618 22:43:40.505980 26765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:40.594058 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 26703
2026-06-18T22:43:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1858 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:43:40 dist-test-slave-lnj3 krb5kdc[26802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:43:40 dist-test-slave-lnj3 krb5kdc[26802](info): set up 2 sockets
Jun 18 22:43:40 dist-test-slave-lnj3 krb5kdc[26802](info): commencing operation
krb5kdc: starting...
W20260618 22:43:41.594776 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:43:41 dist-test-slave-lnj3 krb5kdc[26802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:43:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:43:41.829685 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39209
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:42.148020 26815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:42.148324 26815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:42.148442 26815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:42.155642 26815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:43:42.155751 26815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:42.155807 26815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:42.155885 26815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:43:42.155948 26815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:43:42.165925 26815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:42.167279 26815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:42.168534 26815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:42.180157 26820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:42.180074 26821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:42.180123 26823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:42.180822 26815 server_base.cc:1034] running on GCE node
I20260618 22:43:42.181566 26815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:42.183012 26815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:42.184226 26815 hybrid_clock.cc:648] HybridClock initialized: now 1781822622184165 us; error 46 us; skew 500 ppm
I20260618 22:43:42.184545 26815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:42 dist-test-slave-lnj3 krb5kdc[26802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:42.187580 26815 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:43:42.188956 26815 webserver.cc:460] Webserver started at http://127.0.0.1:43029/ using document root <none> and password file <none>
I20260618 22:43:42.189615 26815 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:42.189731 26815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:42.190006 26815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:42.192981 26815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "7fb9965d1c52473e86c37845f3987927"
format_stamp: "Formatted at 2026-06-18 22:43:42 on dist-test-slave-lnj3"
server_key: "322cc58162b422147789b7a7b7e731a1"
server_key_iv: "42a806f5dc9306fc697c9543ba093f94"
server_key_version: "encryptionkey@0"
I20260618 22:43:42.193718 26815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "7fb9965d1c52473e86c37845f3987927"
format_stamp: "Formatted at 2026-06-18 22:43:42 on dist-test-slave-lnj3"
server_key: "322cc58162b422147789b7a7b7e731a1"
server_key_iv: "42a806f5dc9306fc697c9543ba093f94"
server_key_version: "encryptionkey@0"
I20260618 22:43:42.198752 26815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260618 22:43:42.202203 26830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:42.203817 26815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:43:42.203996 26815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "7fb9965d1c52473e86c37845f3987927"
format_stamp: "Formatted at 2026-06-18 22:43:42 on dist-test-slave-lnj3"
server_key: "322cc58162b422147789b7a7b7e731a1"
server_key_iv: "42a806f5dc9306fc697c9543ba093f94"
server_key_version: "encryptionkey@0"
I20260618 22:43:42.204198 26815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:42.229401 26815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:42.233659 26815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:42.233924 26815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:42.250144 26815 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39209
I20260618 22:43:42.250156 26881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39209 every 8 connection(s)
I20260618 22:43:42.251510 26815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:43:42.252277 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 26815
I20260618 22:43:42.252446 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:43:42.252938 5975 external_mini_cluster.cc:1428] Setting key 1806efab489e083e5da39d8d9dcd1b8b
I20260618 22:43:42.255678 26882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:42.265977 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:42.254716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33085 (local address 127.0.0.1:39209)
0618 22:43:42.255456 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:42.255468 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:42.255506 (+ 38us) server_negotiation.cc:407] Connection header received
0618 22:43:42.257130 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:42.257152 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:42.257534 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:42.257886 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:42.258474 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:42.259803 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:42.261627 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:42.262294 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:42.262416 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:42.263567 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:42.263575 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:42.263588 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:42.264124 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:42.264194 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:42.264225 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:42.264311 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:42.264596 (+ 285us) server_negotiation.cc:299] Negotiation successful
0618 22:43:42.264885 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":322,"threads_started":1}
I20260618 22:43:42.267474 26882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927: Bootstrap starting.
I20260618 22:43:42.269805 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:43:42.270906 26882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:42.272070 26882 log.cc:824] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:42.275102 26882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927: No bootstrap required, opened a new log
I20260618 22:43:42.280301 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:42.274688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59709 (local address 127.0.0.1:39209)
0618 22:43:42.274842 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:42.274848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:42.275250 (+ 402us) server_negotiation.cc:407] Connection header received
0618 22:43:42.275418 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:42.275423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:42.275483 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:42.275568 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:42.276030 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:42.276781 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:42.278050 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:42.278492 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:42.278596 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:42.279615 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:42.279620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:42.279624 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:42.279641 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:42.279680 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:42.279695 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:42.279773 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:42.279915 (+ 142us) server_negotiation.cc:299] Negotiation successful
0618 22:43:42.280005 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 22:43:42.282859 26882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fb9965d1c52473e86c37845f3987927" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39209 } }
I20260618 22:43:42.283243 26882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:42.283321 26882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb9965d1c52473e86c37845f3987927, State: Initialized, Role: FOLLOWER
I20260618 22:43:42.283874 26882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fb9965d1c52473e86c37845f3987927" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39209 } }
I20260618 22:43:42.284071 26882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:43:42.284166 26882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:43:42.284312 26882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:42.286406 26882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fb9965d1c52473e86c37845f3987927" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39209 } }
I20260618 22:43:42.286893 26882 leader_election.cc:302] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fb9965d1c52473e86c37845f3987927; no voters:
I20260618 22:43:42.287497 26882 leader_election.cc:288] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:43:42.287647 26892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:42.288616 26892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927 [term 1 LEADER]: Becoming Leader. State: Replica: 7fb9965d1c52473e86c37845f3987927, State: Running, Role: LEADER
I20260618 22:43:42.289201 26892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fb9965d1c52473e86c37845f3987927" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39209 } }
I20260618 22:43:42.289796 26882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:43:42.291688 26894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fb9965d1c52473e86c37845f3987927. Latest consensus state: current_term: 1 leader_uuid: "7fb9965d1c52473e86c37845f3987927" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fb9965d1c52473e86c37845f3987927" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39209 } } }
I20260618 22:43:42.292016 26894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:42.292812 26897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:43:42.292680 26893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fb9965d1c52473e86c37845f3987927" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fb9965d1c52473e86c37845f3987927" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39209 } } }
I20260618 22:43:42.293010 26893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:42.298663 26897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:43:42.307698 26897 catalog_manager.cc:1347] Generated new cluster ID: b639d17e1be749fc94331c6f00d551b4
I20260618 22:43:42.307824 26897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:43:42.323362 26897 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:43:42.325227 26897 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:43:42.334594 26897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fb9965d1c52473e86c37845f3987927: Generated new TSK 0
I20260618 22:43:42.335439 26897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:43:42.351078 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 26815
W20260618 22:43:42.352206 26877 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33085
W20260618 22:43:42.352916 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 26815: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:43:42.353046 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 26815
2026-06-18T22:43:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1759 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:43:42 dist-test-slave-lnj3 krb5kdc[26913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:43:42 dist-test-slave-lnj3 krb5kdc[26913](info): set up 2 sockets
Jun 18 22:43:42 dist-test-slave-lnj3 krb5kdc[26913](info): commencing operation
krb5kdc: starting...
W20260618 22:43:43.366850 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:43:43 dist-test-slave-lnj3 krb5kdc[26913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:43:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:43:43.609884 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37675
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:43.932687 26926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:43.933000 26926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:43.933117 26926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:43.940518 26926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:43:43.940644 26926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:43.940696 26926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:43.940786 26926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:43:43.940842 26926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:43:43.950913 26926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:43.952214 26926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:43.953464 26926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:43.965119 26931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:43.965051 26932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:43.965082 26934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:43.965695 26926 server_base.cc:1034] running on GCE node
I20260618 22:43:43.966586 26926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:43.968041 26926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:43.969251 26926 hybrid_clock.cc:648] HybridClock initialized: now 1781822623969206 us; error 32 us; skew 500 ppm
I20260618 22:43:43.969586 26926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:43 dist-test-slave-lnj3 krb5kdc[26913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:43.979861 26926 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:43:43.981850 26926 webserver.cc:460] Webserver started at http://127.0.0.1:35709/ using document root <none> and password file <none>
I20260618 22:43:43.982815 26926 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:43.982985 26926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:43.983345 26926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:43.987488 26926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "618025a3a3734c83ae8f0f849f30d726"
format_stamp: "Formatted at 2026-06-18 22:43:43 on dist-test-slave-lnj3"
server_key: "9a9f8e5a24a867b5d8e422ea90ee9a8c"
server_key_iv: "563c8c74f103fce720fba526ea6b1c6f"
server_key_version: "encryptionkey@0"
I20260618 22:43:43.988514 26926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "618025a3a3734c83ae8f0f849f30d726"
format_stamp: "Formatted at 2026-06-18 22:43:43 on dist-test-slave-lnj3"
server_key: "9a9f8e5a24a867b5d8e422ea90ee9a8c"
server_key_iv: "563c8c74f103fce720fba526ea6b1c6f"
server_key_version: "encryptionkey@0"
I20260618 22:43:43.995054 26926 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260618 22:43:43.998857 26941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:44.000280 26926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:43:44.000456 26926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "618025a3a3734c83ae8f0f849f30d726"
format_stamp: "Formatted at 2026-06-18 22:43:43 on dist-test-slave-lnj3"
server_key: "9a9f8e5a24a867b5d8e422ea90ee9a8c"
server_key_iv: "563c8c74f103fce720fba526ea6b1c6f"
server_key_version: "encryptionkey@0"
I20260618 22:43:44.000653 26926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:44.006574 26926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:44.010475 26926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:44.010776 26926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:44.027374 26926 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37675
I20260618 22:43:44.027410 26992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37675 every 8 connection(s)
I20260618 22:43:44.028687 26926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:43:44.032230 26993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:44.033094 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 26926
I20260618 22:43:44.033250 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:43:44.033635 5975 external_mini_cluster.cc:1428] Setting key b0b5a4700e824d9ff2ce08c0bac4b0a6
I20260618 22:43:44.042387 26996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:44.035394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54861 (local address 127.0.0.1:37675)
0618 22:43:44.036168 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:44.036183 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:44.036235 (+ 52us) server_negotiation.cc:407] Connection header received
0618 22:43:44.038788 (+ 2553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:44.038818 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:44.039274 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:44.039723 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:44.040590 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:44.040599 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:44.040615 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:44.040630 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:44.040689 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:44.040729 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:44.040925 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:44.041046 (+ 121us) server_negotiation.cc:299] Negotiation successful
0618 22:43:44.041231 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":327,"threads_started":1}
I20260618 22:43:44.045928 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:43:44.046074 26993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726: Bootstrap starting.
I20260618 22:43:44.049021 26993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:44.050122 26993 log.cc:824] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:44.052273 26996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:44.050026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41529 (local address 127.0.0.1:37675)
0618 22:43:44.050207 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:44.050213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:44.050500 (+ 287us) server_negotiation.cc:407] Connection header received
0618 22:43:44.050681 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:44.050685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:44.050735 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:44.050831 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:44.051672 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:44.051678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:44.051683 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:44.051701 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:44.051738 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:44.051755 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:44.051848 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:44.051955 (+ 107us) server_negotiation.cc:299] Negotiation successful
0618 22:43:44.052016 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260618 22:43:44.052960 26993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726: No bootstrap required, opened a new log
I20260618 22:43:44.060487 26993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "618025a3a3734c83ae8f0f849f30d726" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37675 } }
I20260618 22:43:44.060878 26993 raft_consensus.cc:374] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:44.060968 26993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 618025a3a3734c83ae8f0f849f30d726, State: Initialized, Role: FOLLOWER
I20260618 22:43:44.061612 26993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "618025a3a3734c83ae8f0f849f30d726" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37675 } }
I20260618 22:43:44.061834 26993 raft_consensus.cc:388] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:43:44.061962 26993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:43:44.062103 26993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:44.064334 26993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "618025a3a3734c83ae8f0f849f30d726" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37675 } }
I20260618 22:43:44.064848 26993 leader_election.cc:302] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 618025a3a3734c83ae8f0f849f30d726; no voters:
I20260618 22:43:44.065438 26993 leader_election.cc:288] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:43:44.065539 27003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:44.066118 27003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726 [term 1 LEADER]: Becoming Leader. State: Replica: 618025a3a3734c83ae8f0f849f30d726, State: Running, Role: LEADER
I20260618 22:43:44.066731 27003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "618025a3a3734c83ae8f0f849f30d726" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37675 } }
I20260618 22:43:44.067643 26993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:43:44.069716 27004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "618025a3a3734c83ae8f0f849f30d726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "618025a3a3734c83ae8f0f849f30d726" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37675 } } }
I20260618 22:43:44.070151 27004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:44.070397 27005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 618025a3a3734c83ae8f0f849f30d726. Latest consensus state: current_term: 1 leader_uuid: "618025a3a3734c83ae8f0f849f30d726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "618025a3a3734c83ae8f0f849f30d726" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37675 } } }
I20260618 22:43:44.070708 27005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:44.072547 27011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:43:44.077852 27011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:43:44.086869 27011 catalog_manager.cc:1347] Generated new cluster ID: 330e3ee662904a87bc888e91d2dc91ad
I20260618 22:43:44.086984 27011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:43:44.110291 27011 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:43:44.110749 27011 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:43:44.116919 27011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 618025a3a3734c83ae8f0f849f30d726: Generated new TSK 0
I20260618 22:43:44.117753 27011 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:43:44.131615 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 26926
W20260618 22:43:44.132875 26988 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54861
W20260618 22:43:44.133533 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 26926: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:43:44.133692 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 26926
2026-06-18T22:43:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1780 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:43:44 dist-test-slave-lnj3 krb5kdc[27024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:43:44 dist-test-slave-lnj3 krb5kdc[27024](info): set up 2 sockets
Jun 18 22:43:44 dist-test-slave-lnj3 krb5kdc[27024](info): commencing operation
krb5kdc: starting...
W20260618 22:43:45.150869 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:43:45 dist-test-slave-lnj3 krb5kdc[27024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:43:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:43:45.391394 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35111
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:45.720261 27037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:45.720567 27037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:45.720717 27037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:45.727924 27037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:43:45.728055 27037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:45.728137 27037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:45.728224 27037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:43:45.728281 27037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:43:45.738147 27037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:45.739449 27037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:45.740715 27037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:45.751685 27042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:45.751725 27043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:45.751762 27045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:45.752506 27037 server_base.cc:1034] running on GCE node
I20260618 22:43:45.753278 27037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:45.754686 27037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:45.755913 27037 hybrid_clock.cc:648] HybridClock initialized: now 1781822625755860 us; error 47 us; skew 500 ppm
I20260618 22:43:45.756256 27037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:45 dist-test-slave-lnj3 krb5kdc[27024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:45.759321 27037 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:43:45.760847 27037 webserver.cc:460] Webserver started at http://127.0.0.1:34725/ using document root <none> and password file <none>
I20260618 22:43:45.761514 27037 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:45.761629 27037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:45.761917 27037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:45.764969 27037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "834721288b304645a9695edd0cf7cb4c"
format_stamp: "Formatted at 2026-06-18 22:43:45 on dist-test-slave-lnj3"
server_key: "b1b3c094520d3968edca679759022e96"
server_key_iv: "ab4025a0d58b8c397e6c14fbf25a70c8"
server_key_version: "encryptionkey@0"
I20260618 22:43:45.765724 27037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "834721288b304645a9695edd0cf7cb4c"
format_stamp: "Formatted at 2026-06-18 22:43:45 on dist-test-slave-lnj3"
server_key: "b1b3c094520d3968edca679759022e96"
server_key_iv: "ab4025a0d58b8c397e6c14fbf25a70c8"
server_key_version: "encryptionkey@0"
I20260618 22:43:45.770999 27037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260618 22:43:45.774533 27052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:45.776000 27037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:43:45.776166 27037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "834721288b304645a9695edd0cf7cb4c"
format_stamp: "Formatted at 2026-06-18 22:43:45 on dist-test-slave-lnj3"
server_key: "b1b3c094520d3968edca679759022e96"
server_key_iv: "ab4025a0d58b8c397e6c14fbf25a70c8"
server_key_version: "encryptionkey@0"
I20260618 22:43:45.776361 27037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:45.792541 27037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:45.796820 27037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:45.797086 27037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:45.814620 27037 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35111
I20260618 22:43:45.814671 27103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35111 every 8 connection(s)
I20260618 22:43:45.816140 27037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:43:45.819854 27104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:45.825582 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 27037
I20260618 22:43:45.825753 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:43:45.826159 5975 external_mini_cluster.cc:1428] Setting key 9b99eabe78271342c7e04dbd732804bc
I20260618 22:43:45.830319 27104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c: Bootstrap starting.
I20260618 22:43:45.833885 27104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:45.835150 27104 log.cc:824] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:45.838470 27104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c: No bootstrap required, opened a new log
Jun 18 22:43:45 dist-test-slave-lnj3 krb5kdc[27024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260618 22:43:45.847879 27104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834721288b304645a9695edd0cf7cb4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35111 } }
I20260618 22:43:45.848400 27104 raft_consensus.cc:374] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:45.848521 27104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 834721288b304645a9695edd0cf7cb4c, State: Initialized, Role: FOLLOWER
I20260618 22:43:45.849207 27104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834721288b304645a9695edd0cf7cb4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35111 } }
I20260618 22:43:45.849434 27104 raft_consensus.cc:388] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:43:45.849572 27104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:43:45.849735 27104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:45.850216 27107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:45.827883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:59327 (local address 127.0.0.1:35111)
0618 22:43:45.828616 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:45.828626 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:45.828666 (+ 40us) server_negotiation.cc:407] Connection header received
0618 22:43:45.830277 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:45.830299 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:45.830773 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:45.831088 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:45.832825 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:45.834171 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:45.835964 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:45.836875 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:45.837001 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:45.841758 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:45.841771 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:45.841790 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:45.841827 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:45.845009 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:45.846019 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:45.846027 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:45.846037 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:45.846240 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:45.847052 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:45.847056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:45.847058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:45.847742 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:45.848001 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:45.848668 (+ 667us) server_negotiation.cc:299] Negotiation successful
0618 22:43:45.849044 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":326,"threads_started":1}
I20260618 22:43:45.852255 27104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834721288b304645a9695edd0cf7cb4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35111 } }
I20260618 22:43:45.852807 27104 leader_election.cc:302] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 834721288b304645a9695edd0cf7cb4c; no voters:
I20260618 22:43:45.853448 27104 leader_election.cc:288] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:43:45.853554 27109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:45.854420 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:43:45.855381 27104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:43:45.859879 27109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c [term 1 LEADER]: Becoming Leader. State: Replica: 834721288b304645a9695edd0cf7cb4c, State: Running, Role: LEADER
I20260618 22:43:45.860558 27109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834721288b304645a9695edd0cf7cb4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35111 } }
I20260618 22:43:45.863647 27110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "834721288b304645a9695edd0cf7cb4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834721288b304645a9695edd0cf7cb4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35111 } } }
I20260618 22:43:45.863994 27110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:45.864444 27111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 834721288b304645a9695edd0cf7cb4c. Latest consensus state: current_term: 1 leader_uuid: "834721288b304645a9695edd0cf7cb4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834721288b304645a9695edd0cf7cb4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35111 } } }
I20260618 22:43:45.864727 27111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:45.873039 27121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:43:45.877686 27107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:45.860648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:35003 (local address 127.0.0.1:35111)
0618 22:43:45.860826 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:45.860832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:45.860852 (+ 20us) server_negotiation.cc:407] Connection header received
0618 22:43:45.860984 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:45.860989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:45.861067 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:45.861136 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:45.863126 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:45.864318 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:45.866287 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:45.867044 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:45.867138 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:45.870600 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:45.870607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:45.870613 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:45.870695 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:45.874110 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:45.874899 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:45.874905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:45.874908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:45.875064 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:45.875668 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:45.875673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:45.875676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:45.876180 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:45.876426 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:45.877110 (+ 684us) server_negotiation.cc:299] Negotiation successful
0618 22:43:45.877311 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260618 22:43:45.879769 27121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:43:45.889115 27121 catalog_manager.cc:1347] Generated new cluster ID: 87b148d0c1914d1097c5a497e94098a7
I20260618 22:43:45.889230 27121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:43:45.904583 27121 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:43:45.906383 27121 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:43:45.914880 27121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 834721288b304645a9695edd0cf7cb4c: Generated new TSK 0
I20260618 22:43:45.915680 27121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:43:45.922784 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 27037
I20260618 22:43:45.924284 27099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:46.022864 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 27037
W20260618 22:43:46.036405 27022 connection.cc:535] client connection to 127.0.0.1:35111 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35111): Connection reset by peer (error 104)
2026-06-18T22:43:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1888 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:43:46 dist-test-slave-lnj3 krb5kdc[27137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:43:46 dist-test-slave-lnj3 krb5kdc[27137](info): set up 2 sockets
Jun 18 22:43:46 dist-test-slave-lnj3 krb5kdc[27137](info): commencing operation
krb5kdc: starting...
W20260618 22:43:47.042826 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:43:47 dist-test-slave-lnj3 krb5kdc[27137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:43:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:43:47.286000 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37099
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:47.629384 27150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:47.629696 27150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:47.629833 27150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:47.637121 27150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:43:47.637253 27150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:47.637332 27150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:47.637415 27150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:43:47.637475 27150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:43:47.647503 27150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:47.648851 27150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:47.650172 27150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:47.661655 27155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:47.661616 27156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:47.661711 27158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:47.662379 27150 server_base.cc:1034] running on GCE node
I20260618 22:43:47.663239 27150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:47.664664 27150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:47.665864 27150 hybrid_clock.cc:648] HybridClock initialized: now 1781822627665814 us; error 44 us; skew 500 ppm
I20260618 22:43:47.666209 27150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:47 dist-test-slave-lnj3 krb5kdc[27137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:47.669345 27150 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:43:47.670898 27150 webserver.cc:460] Webserver started at http://127.0.0.1:46599/ using document root <none> and password file <none>
I20260618 22:43:47.671581 27150 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:47.671697 27150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:47.671979 27150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:47.674984 27150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "1f2f7ec84762472c84fe3486a8861b08"
format_stamp: "Formatted at 2026-06-18 22:43:47 on dist-test-slave-lnj3"
server_key: "ea9eac7e55a1726551e63a2e38c869f6"
server_key_iv: "8667d4fa0e9032215c6c09c9e1760447"
server_key_version: "encryptionkey@0"
I20260618 22:43:47.675731 27150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "1f2f7ec84762472c84fe3486a8861b08"
format_stamp: "Formatted at 2026-06-18 22:43:47 on dist-test-slave-lnj3"
server_key: "ea9eac7e55a1726551e63a2e38c869f6"
server_key_iv: "8667d4fa0e9032215c6c09c9e1760447"
server_key_version: "encryptionkey@0"
I20260618 22:43:47.681066 27150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260618 22:43:47.684824 27165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:47.686338 27150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:43:47.686520 27150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "1f2f7ec84762472c84fe3486a8861b08"
format_stamp: "Formatted at 2026-06-18 22:43:47 on dist-test-slave-lnj3"
server_key: "ea9eac7e55a1726551e63a2e38c869f6"
server_key_iv: "8667d4fa0e9032215c6c09c9e1760447"
server_key_version: "encryptionkey@0"
I20260618 22:43:47.686743 27150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:47.714850 27150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:47.719206 27150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:47.719494 27150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:47.737870 27150 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37099
I20260618 22:43:47.737890 27216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37099 every 8 connection(s)
I20260618 22:43:47.739435 27150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:43:47.743165 27217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:47.749257 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 27150
I20260618 22:43:47.749421 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:43:47.749840 5975 external_mini_cluster.cc:1428] Setting key c0b486547f8b584f7bcc100412e243dc
I20260618 22:43:47.754359 27217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08: Bootstrap starting.
I20260618 22:43:47.758306 27217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:47.759356 27217 log.cc:824] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:47.762465 27217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08: No bootstrap required, opened a new log
I20260618 22:43:47.764422 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:47.751684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:38755 (local address 127.0.0.1:37099)
0618 22:43:47.752442 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:47.752457 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:47.752507 (+ 50us) server_negotiation.cc:407] Connection header received
0618 22:43:47.754804 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:47.754835 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:47.755375 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:47.755815 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:47.756381 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:47.758294 (+ 1913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:47.759635 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:47.760469 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:47.760644 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:47.761707 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:47.761728 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:47.761746 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:47.762438 (+ 692us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:47.762539 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:47.762570 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:47.762750 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:47.762956 (+ 206us) server_negotiation.cc:299] Negotiation successful
0618 22:43:47.763138 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":297,"threads_started":1}
I20260618 22:43:47.768216 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:43:47.770699 27217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f2f7ec84762472c84fe3486a8861b08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37099 } }
I20260618 22:43:47.771126 27217 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:47.771237 27217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f2f7ec84762472c84fe3486a8861b08, State: Initialized, Role: FOLLOWER
I20260618 22:43:47.771814 27217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f2f7ec84762472c84fe3486a8861b08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37099 } }
I20260618 22:43:47.772039 27217 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:43:47.772229 27217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:43:47.772436 27217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:47.775419 27217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f2f7ec84762472c84fe3486a8861b08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37099 } }
I20260618 22:43:47.775902 27217 leader_election.cc:302] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f2f7ec84762472c84fe3486a8861b08; no voters:
I20260618 22:43:47.776543 27217 leader_election.cc:288] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:43:47.776638 27227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:47.777832 27227 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08 [term 1 LEADER]: Becoming Leader. State: Replica: 1f2f7ec84762472c84fe3486a8861b08, State: Running, Role: LEADER
I20260618 22:43:47.778445 27227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f2f7ec84762472c84fe3486a8861b08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37099 } }
I20260618 22:43:47.778863 27217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:43:47.779839 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:47.772355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:36339 (local address 127.0.0.1:37099)
0618 22:43:47.772539 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:47.772544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:47.772827 (+ 283us) server_negotiation.cc:407] Connection header received
0618 22:43:47.772959 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:47.772964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:47.773050 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:47.773162 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:47.773606 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:47.774704 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:47.776266 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:47.777043 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:47.777178 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:47.778746 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:47.778753 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:47.778757 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:47.778778 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:47.778849 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:47.778872 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:47.778983 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:47.779437 (+ 454us) server_negotiation.cc:299] Negotiation successful
0618 22:43:47.779487 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260618 22:43:47.782195 27229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f2f7ec84762472c84fe3486a8861b08. Latest consensus state: current_term: 1 leader_uuid: "1f2f7ec84762472c84fe3486a8861b08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f2f7ec84762472c84fe3486a8861b08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37099 } } }
I20260618 22:43:47.782644 27229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:47.783161 27228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f2f7ec84762472c84fe3486a8861b08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f2f7ec84762472c84fe3486a8861b08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37099 } } }
I20260618 22:43:47.783442 27228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:47.783622 27233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:43:47.795171 27233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:43:47.805447 27233 catalog_manager.cc:1347] Generated new cluster ID: ee6b6f45bd5a4281a7ed31bc07eaae8c
I20260618 22:43:47.805570 27233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:43:47.816056 27233 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:43:47.818013 27233 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:43:47.826942 27233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f2f7ec84762472c84fe3486a8861b08: Generated new TSK 0
I20260618 22:43:47.827791 27233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:43:47.838771 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 27150
W20260618 22:43:47.839917 27212 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.18:38755
W20260618 22:43:47.840677 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 27150: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:43:47.840812 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 27150
W20260618 22:43:47.854168 27135 connection.cc:535] client connection to 127.0.0.1:37099 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37099): Connection reset by peer (error 104)
2026-06-18T22:43:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1817 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:43:47 dist-test-slave-lnj3 krb5kdc[27249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:43:47 dist-test-slave-lnj3 krb5kdc[27249](info): set up 2 sockets
Jun 18 22:43:47 dist-test-slave-lnj3 krb5kdc[27249](info): commencing operation
krb5kdc: starting...
W20260618 22:43:48.850864 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:43:49 dist-test-slave-lnj3 krb5kdc[27249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:43:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:43:49.086078 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40487
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:49.409533 27262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:49.409866 27262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:49.410009 27262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:49.417348 27262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:43:49.417474 27262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:49.417552 27262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:49.417639 27262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:43:49.417694 27262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:43:49.427500 27262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:49.428754 27262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:49.430016 27262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:49.441042 27268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:49.441107 27267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:49.441128 27270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:49.441926 27262 server_base.cc:1034] running on GCE node
I20260618 22:43:49.442787 27262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:49.444160 27262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:49.445384 27262 hybrid_clock.cc:648] HybridClock initialized: now 1781822629445336 us; error 41 us; skew 500 ppm
I20260618 22:43:49.445729 27262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:49 dist-test-slave-lnj3 krb5kdc[27249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:49.451880 27262 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:43:49.453361 27262 webserver.cc:460] Webserver started at http://127.0.0.1:42195/ using document root <none> and password file <none>
I20260618 22:43:49.454020 27262 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:49.454123 27262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:49.454370 27262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:49.457334 27262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "6b660418f4934ec08a24b0aa38c9ffdb"
format_stamp: "Formatted at 2026-06-18 22:43:49 on dist-test-slave-lnj3"
server_key: "cc8fb447fcbebb5ea8b700989a80844d"
server_key_iv: "0e589f877a174fed85980c7a33a573e0"
server_key_version: "encryptionkey@0"
I20260618 22:43:49.458042 27262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "6b660418f4934ec08a24b0aa38c9ffdb"
format_stamp: "Formatted at 2026-06-18 22:43:49 on dist-test-slave-lnj3"
server_key: "cc8fb447fcbebb5ea8b700989a80844d"
server_key_iv: "0e589f877a174fed85980c7a33a573e0"
server_key_version: "encryptionkey@0"
I20260618 22:43:49.463191 27262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260618 22:43:49.466667 27277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:49.468152 27262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:43:49.468309 27262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "6b660418f4934ec08a24b0aa38c9ffdb"
format_stamp: "Formatted at 2026-06-18 22:43:49 on dist-test-slave-lnj3"
server_key: "cc8fb447fcbebb5ea8b700989a80844d"
server_key_iv: "0e589f877a174fed85980c7a33a573e0"
server_key_version: "encryptionkey@0"
I20260618 22:43:49.468497 27262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:49.474529 27262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:49.478927 27262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:49.479179 27262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:49.496718 27262 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40487
I20260618 22:43:49.496745 27328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40487 every 8 connection(s)
I20260618 22:43:49.498118 27262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:43:49.498446 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 27262
I20260618 22:43:49.498602 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:43:49.499143 5975 external_mini_cluster.cc:1428] Setting key e6a59e6dd6949174829d2ab2b0aaae67
I20260618 22:43:49.502408 27329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:49.506510 27332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:49.500929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:53131 (local address 127.0.0.1:40487)
0618 22:43:49.501659 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:49.501670 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:49.501710 (+ 40us) server_negotiation.cc:407] Connection header received
0618 22:43:49.503360 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:49.503382 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:49.503699 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:49.504056 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:49.505053 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:49.505062 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:49.505073 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:49.505086 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:49.505154 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:49.505176 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:49.505331 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:49.505438 (+ 107us) server_negotiation.cc:299] Negotiation successful
0618 22:43:49.505606 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":331,"threads_started":1}
I20260618 22:43:49.510046 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:43:49.513978 27329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb: Bootstrap starting.
I20260618 22:43:49.517602 27329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:49.518332 27332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:49.515963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:36437 (local address 127.0.0.1:40487)
0618 22:43:49.516129 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:49.516135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:49.516509 (+ 374us) server_negotiation.cc:407] Connection header received
0618 22:43:49.516703 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:49.516708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:49.516772 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:49.516850 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:49.517702 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:49.517708 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:49.517721 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:49.517739 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:49.517785 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:49.517804 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:49.517922 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:49.518012 (+ 90us) server_negotiation.cc:299] Negotiation successful
0618 22:43:49.518078 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:43:49.518877 27329 log.cc:824] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:49.521625 27329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb: No bootstrap required, opened a new log
I20260618 22:43:49.529678 27329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b660418f4934ec08a24b0aa38c9ffdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40487 } }
I20260618 22:43:49.530072 27329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:49.530155 27329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b660418f4934ec08a24b0aa38c9ffdb, State: Initialized, Role: FOLLOWER
I20260618 22:43:49.530773 27329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b660418f4934ec08a24b0aa38c9ffdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40487 } }
I20260618 22:43:49.530983 27329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:43:49.531081 27329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:43:49.531208 27329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:49.533339 27329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b660418f4934ec08a24b0aa38c9ffdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40487 } }
I20260618 22:43:49.533839 27329 leader_election.cc:302] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b660418f4934ec08a24b0aa38c9ffdb; no voters:
I20260618 22:43:49.534363 27329 leader_election.cc:288] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:43:49.534503 27339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:49.535544 27339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb [term 1 LEADER]: Becoming Leader. State: Replica: 6b660418f4934ec08a24b0aa38c9ffdb, State: Running, Role: LEADER
I20260618 22:43:49.536067 27339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b660418f4934ec08a24b0aa38c9ffdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40487 } }
I20260618 22:43:49.536551 27329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:43:49.538913 27341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b660418f4934ec08a24b0aa38c9ffdb. Latest consensus state: current_term: 1 leader_uuid: "6b660418f4934ec08a24b0aa38c9ffdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b660418f4934ec08a24b0aa38c9ffdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40487 } } }
I20260618 22:43:49.538939 27340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b660418f4934ec08a24b0aa38c9ffdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b660418f4934ec08a24b0aa38c9ffdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40487 } } }
I20260618 22:43:49.539386 27340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:49.539388 27341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:49.540890 27344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:43:49.545572 27344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:43:49.553445 27344 catalog_manager.cc:1347] Generated new cluster ID: fafed47417424c20912dba6f8a528d3e
I20260618 22:43:49.553555 27344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:43:49.569295 27344 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:43:49.569692 27344 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:43:49.586825 27344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b660418f4934ec08a24b0aa38c9ffdb: Generated new TSK 0
I20260618 22:43:49.587657 27344 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:43:49.592283 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 27262
W20260618 22:43:49.593376 27324 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.18:53131
W20260618 22:43:49.594107 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 27262: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:43:49.594255 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 27262
2026-06-18T22:43:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1753 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:43:49 dist-test-slave-lnj3 krb5kdc[27360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:43:49 dist-test-slave-lnj3 krb5kdc[27360](info): set up 2 sockets
Jun 18 22:43:49 dist-test-slave-lnj3 krb5kdc[27360](info): commencing operation
krb5kdc: starting...
W20260618 22:43:50.603403 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:43:50 dist-test-slave-lnj3 krb5kdc[27360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:43:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:43:50.845933 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44689
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:51.167912 27373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:51.168205 27373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:51.168322 27373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:51.175623 27373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:43:51.175740 27373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:51.175809 27373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:51.175882 27373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:43:51.175943 27373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:43:51.185792 27373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44689
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:51.187076 27373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:51.188378 27373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:51.199312 27379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:51.199358 27378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:51.199354 27381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:51.200965 27373 server_base.cc:1034] running on GCE node
I20260618 22:43:51.201704 27373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:51.203097 27373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:51.204291 27373 hybrid_clock.cc:648] HybridClock initialized: now 1781822631204246 us; error 33 us; skew 500 ppm
I20260618 22:43:51.204612 27373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:51 dist-test-slave-lnj3 krb5kdc[27360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:51.207640 27373 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:43:51.209004 27373 webserver.cc:460] Webserver started at http://127.0.0.1:41539/ using document root <none> and password file <none>
I20260618 22:43:51.209698 27373 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:51.209813 27373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:51.210093 27373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:51.213161 27373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "b9d5cbbe69494e45a9b5d8db69fda719"
format_stamp: "Formatted at 2026-06-18 22:43:51 on dist-test-slave-lnj3"
server_key: "c6eea6e00a2b64afb7f83ca1b9dece38"
server_key_iv: "a67a755604ee8d1101c342ccb66c2d13"
server_key_version: "encryptionkey@0"
I20260618 22:43:51.213907 27373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "b9d5cbbe69494e45a9b5d8db69fda719"
format_stamp: "Formatted at 2026-06-18 22:43:51 on dist-test-slave-lnj3"
server_key: "c6eea6e00a2b64afb7f83ca1b9dece38"
server_key_iv: "a67a755604ee8d1101c342ccb66c2d13"
server_key_version: "encryptionkey@0"
I20260618 22:43:51.219101 27373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260618 22:43:51.222599 27388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:51.223980 27373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:43:51.224136 27373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "b9d5cbbe69494e45a9b5d8db69fda719"
format_stamp: "Formatted at 2026-06-18 22:43:51 on dist-test-slave-lnj3"
server_key: "c6eea6e00a2b64afb7f83ca1b9dece38"
server_key_iv: "a67a755604ee8d1101c342ccb66c2d13"
server_key_version: "encryptionkey@0"
I20260618 22:43:51.224329 27373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:51.252979 27373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:51.257330 27373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:51.257593 27373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:51.274520 27373 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44689
I20260618 22:43:51.274566 27439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44689 every 8 connection(s)
I20260618 22:43:51.275954 27373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:43:51.279362 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 27373
I20260618 22:43:51.279561 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:43:51.280040 27440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:51.280103 5975 external_mini_cluster.cc:1428] Setting key ecc48cca20014e859dd2168b93f4e412
I20260618 22:43:51.294413 27440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719: Bootstrap starting.
Jun 18 22:43:51 dist-test-slave-lnj3 krb5kdc[27360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260618 22:43:51.297868 27440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:51.299048 27440 log.cc:824] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:51.301805 27440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719: No bootstrap required, opened a new log
I20260618 22:43:51.302980 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:51.281902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:49121 (local address 127.0.0.1:44689)
0618 22:43:51.282708 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:51.282722 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:51.282766 (+ 44us) server_negotiation.cc:407] Connection header received
0618 22:43:51.284696 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:51.284724 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:51.285174 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:51.285545 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:51.287343 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:51.288611 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:51.290317 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:51.291071 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:51.291205 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:51.295836 (+ 4631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:51.295845 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:51.295863 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:51.295909 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:51.298893 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:51.299685 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:51.299695 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:51.299702 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:51.299876 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:51.300307 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:51.300310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:51.300312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:51.300821 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:51.301100 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:51.301577 (+ 477us) server_negotiation.cc:299] Negotiation successful
0618 22:43:51.301900 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":341,"threads_started":1}
I20260618 22:43:51.306676 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:43:51.309552 27440 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d5cbbe69494e45a9b5d8db69fda719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44689 } }
I20260618 22:43:51.309906 27440 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:51.309991 27440 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d5cbbe69494e45a9b5d8db69fda719, State: Initialized, Role: FOLLOWER
I20260618 22:43:51.310599 27440 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d5cbbe69494e45a9b5d8db69fda719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44689 } }
I20260618 22:43:51.310938 27440 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:43:51.311074 27440 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:43:51.311226 27440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:51.313668 27440 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d5cbbe69494e45a9b5d8db69fda719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44689 } }
I20260618 22:43:51.314244 27440 leader_election.cc:302] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9d5cbbe69494e45a9b5d8db69fda719; no voters:
I20260618 22:43:51.314855 27440 leader_election.cc:288] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:43:51.314950 27450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:51.316984 27450 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719 [term 1 LEADER]: Becoming Leader. State: Replica: b9d5cbbe69494e45a9b5d8db69fda719, State: Running, Role: LEADER
I20260618 22:43:51.317164 27440 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:43:51.317543 27450 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d5cbbe69494e45a9b5d8db69fda719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44689 } }
I20260618 22:43:51.320497 27451 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9d5cbbe69494e45a9b5d8db69fda719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d5cbbe69494e45a9b5d8db69fda719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44689 } } }
I20260618 22:43:51.320890 27451 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:51.321425 27452 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9d5cbbe69494e45a9b5d8db69fda719. Latest consensus state: current_term: 1 leader_uuid: "b9d5cbbe69494e45a9b5d8db69fda719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d5cbbe69494e45a9b5d8db69fda719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44689 } } }
I20260618 22:43:51.321692 27452 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:51.325284 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:51.310757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:45231 (local address 127.0.0.1:44689)
0618 22:43:51.310932 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:51.310938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:51.311351 (+ 413us) server_negotiation.cc:407] Connection header received
0618 22:43:51.311514 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:51.311519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:51.311594 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:51.311698 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:51.313017 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:51.314033 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:51.315463 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:51.316070 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:51.316145 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:51.318948 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:51.318953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:51.318956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:51.318986 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:51.322295 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:51.323194 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:51.323199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:51.323202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:51.323340 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:51.323797 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:51.323801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:51.323803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:51.324104 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:51.324353 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:51.324789 (+ 436us) server_negotiation.cc:299] Negotiation successful
0618 22:43:51.324954 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:43:51.327206 27456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:43:51.332531 27456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:43:51.342077 27456 catalog_manager.cc:1347] Generated new cluster ID: 104ad367f15d4fdca17987a86b9ecf1a
I20260618 22:43:51.342206 27456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:43:51.391360 27456 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:43:51.392751 27456 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:43:51.402944 27456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9d5cbbe69494e45a9b5d8db69fda719: Generated new TSK 0
I20260618 22:43:51.403779 27456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:43:51.420665 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 27373
I20260618 22:43:51.421721 27435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:51.518111 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 27373
W20260618 22:43:51.534564 27358 connection.cc:535] client connection to 127.0.0.1:44689 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44689): Connection reset by peer (error 104)
2026-06-18T22:43:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1926 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:43:51 dist-test-slave-lnj3 krb5kdc[27472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:43:51 dist-test-slave-lnj3 krb5kdc[27472](info): set up 2 sockets
Jun 18 22:43:51 dist-test-slave-lnj3 krb5kdc[27472](info): commencing operation
krb5kdc: starting...
W20260618 22:43:52.528024 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:43:52 dist-test-slave-lnj3 krb5kdc[27472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:43:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:43:52.767913 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41801
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:53.083691 27485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:53.084000 27485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:53.084131 27485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:53.091307 27485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:43:53.091418 27485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:53.091471 27485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:53.091554 27485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:43:53.091617 27485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:43:53.101372 27485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41801
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:53.102669 27485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:53.103974 27485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:53.115054 27490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:53.114992 27493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:53.115027 27491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:53.116302 27485 server_base.cc:1034] running on GCE node
I20260618 22:43:53.117063 27485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:53.118427 27485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:53.119621 27485 hybrid_clock.cc:648] HybridClock initialized: now 1781822633119576 us; error 33 us; skew 500 ppm
I20260618 22:43:53.120002 27485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:53 dist-test-slave-lnj3 krb5kdc[27472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:53.122972 27485 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:43:53.124399 27485 webserver.cc:460] Webserver started at http://127.0.0.1:45837/ using document root <none> and password file <none>
I20260618 22:43:53.125052 27485 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:53.125149 27485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:53.125398 27485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:53.128450 27485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "012028e72ab7441ab788b500a835ac32"
format_stamp: "Formatted at 2026-06-18 22:43:53 on dist-test-slave-lnj3"
server_key: "5747c0402e0c306c7fbe33991923f81e"
server_key_iv: "0eee43c88eb4ffb200fcab827438c1d5"
server_key_version: "encryptionkey@0"
I20260618 22:43:53.129182 27485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "012028e72ab7441ab788b500a835ac32"
format_stamp: "Formatted at 2026-06-18 22:43:53 on dist-test-slave-lnj3"
server_key: "5747c0402e0c306c7fbe33991923f81e"
server_key_iv: "0eee43c88eb4ffb200fcab827438c1d5"
server_key_version: "encryptionkey@0"
I20260618 22:43:53.134344 27485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260618 22:43:53.137758 27500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:53.139086 27485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260618 22:43:53.139219 27485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "012028e72ab7441ab788b500a835ac32"
format_stamp: "Formatted at 2026-06-18 22:43:53 on dist-test-slave-lnj3"
server_key: "5747c0402e0c306c7fbe33991923f81e"
server_key_iv: "0eee43c88eb4ffb200fcab827438c1d5"
server_key_version: "encryptionkey@0"
I20260618 22:43:53.139374 27485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:53.171608 27485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:53.175480 27485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:53.175756 27485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:53.193250 27485 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41801
I20260618 22:43:53.193282 27551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41801 every 8 connection(s)
I20260618 22:43:53.194571 27485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:43:53.198208 27552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:53.201088 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 27485
I20260618 22:43:53.201262 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:43:53.201679 5975 external_mini_cluster.cc:1428] Setting key 7d6dea6a04261a46559419b33309d234
I20260618 22:43:53.209504 27552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32: Bootstrap starting.
I20260618 22:43:53.212623 27552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:53.213901 27552 log.cc:824] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:53.214479 27555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:53.203477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:51263 (local address 127.0.0.1:41801)
0618 22:43:53.204200 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:53.204211 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:53.204253 (+ 42us) server_negotiation.cc:407] Connection header received
0618 22:43:53.205870 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:53.205893 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:53.206301 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:53.206696 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:53.207247 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:53.208563 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:53.209902 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:53.210554 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:53.210726 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:53.211919 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:53.211927 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:53.211941 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:53.212433 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:53.212503 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:53.212531 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:53.212675 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:53.212993 (+ 318us) server_negotiation.cc:299] Negotiation successful
0618 22:43:53.213194 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":304,"threads_started":1}
I20260618 22:43:53.216982 27552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32: No bootstrap required, opened a new log
I20260618 22:43:53.218430 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:43:53.225095 27552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012028e72ab7441ab788b500a835ac32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41801 } }
I20260618 22:43:53.225617 27552 raft_consensus.cc:374] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:53.225726 27552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012028e72ab7441ab788b500a835ac32, State: Initialized, Role: FOLLOWER
I20260618 22:43:53.226410 27552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012028e72ab7441ab788b500a835ac32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41801 } }
I20260618 22:43:53.226812 27552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:43:53.226953 27552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:43:53.227120 27552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:53.229249 27555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:53.222665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:56977 (local address 127.0.0.1:41801)
0618 22:43:53.222864 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:53.222874 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:53.223244 (+ 370us) server_negotiation.cc:407] Connection header received
0618 22:43:53.223423 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:53.223427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:53.223494 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:53.223576 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:53.224134 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:53.224878 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:53.226727 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:53.227322 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:53.227419 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:53.228517 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:53.228522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:53.228525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:53.228542 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:53.228584 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:53.228600 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:53.228699 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:53.228894 (+ 195us) server_negotiation.cc:299] Negotiation successful
0618 22:43:53.228951 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 22:43:53.230059 27552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012028e72ab7441ab788b500a835ac32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41801 } }
I20260618 22:43:53.230649 27552 leader_election.cc:302] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 012028e72ab7441ab788b500a835ac32; no voters:
I20260618 22:43:53.231201 27552 leader_election.cc:288] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:43:53.231338 27562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:53.231931 27562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32 [term 1 LEADER]: Becoming Leader. State: Replica: 012028e72ab7441ab788b500a835ac32, State: Running, Role: LEADER
I20260618 22:43:53.232431 27562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012028e72ab7441ab788b500a835ac32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41801 } }
I20260618 22:43:53.233145 27552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:43:53.234992 27564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 012028e72ab7441ab788b500a835ac32. Latest consensus state: current_term: 1 leader_uuid: "012028e72ab7441ab788b500a835ac32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012028e72ab7441ab788b500a835ac32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41801 } } }
I20260618 22:43:53.235360 27564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:53.235813 27563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "012028e72ab7441ab788b500a835ac32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012028e72ab7441ab788b500a835ac32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41801 } } }
I20260618 22:43:53.236085 27563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:53.239486 27569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:43:53.244957 27569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:43:53.254004 27569 catalog_manager.cc:1347] Generated new cluster ID: c7ce70b3152b41cb9712024bf12e8591
I20260618 22:43:53.254127 27569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:43:53.301033 27569 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:43:53.302520 27569 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:43:53.313246 27569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 012028e72ab7441ab788b500a835ac32: Generated new TSK 0
I20260618 22:43:53.314090 27569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:43:53.321924 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 27485
W20260618 22:43:53.323050 27547 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.18:51263
W20260618 22:43:53.323827 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 27485: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:43:53.323980 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 27485
W20260618 22:43:53.337735 27470 connection.cc:535] client connection to 127.0.0.1:41801 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41801): Connection reset by peer (error 104)
2026-06-18T22:43:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1807 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:43:53 dist-test-slave-lnj3 krb5kdc[27583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:43:53 dist-test-slave-lnj3 krb5kdc[27583](info): set up 2 sockets
Jun 18 22:43:53 dist-test-slave-lnj3 krb5kdc[27583](info): commencing operation
krb5kdc: starting...
W20260618 22:43:54.343006 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:43:54 dist-test-slave-lnj3 krb5kdc[27583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:43:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:43:54.582548 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36227
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:54.899356 27596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:54.899658 27596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:54.899794 27596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:54.906903 27596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:43:54.907022 27596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:54.907099 27596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:54.907186 27596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:43:54.907240 27596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:43:54.916965 27596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36227
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:54.918272 27596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:54.919584 27596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:54.931174 27601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:54.931137 27602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:54.931108 27604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:54.932029 27596 server_base.cc:1034] running on GCE node
I20260618 22:43:54.932842 27596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:54.934376 27596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:54.935623 27596 hybrid_clock.cc:648] HybridClock initialized: now 1781822634935556 us; error 50 us; skew 500 ppm
I20260618 22:43:54.936028 27596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:54 dist-test-slave-lnj3 krb5kdc[27583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:54.944887 27596 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:43:54.946408 27596 webserver.cc:460] Webserver started at http://127.0.0.1:41231/ using document root <none> and password file <none>
I20260618 22:43:54.947124 27596 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:54.947240 27596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:54.947523 27596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:54.950436 27596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "7dff1db33ee04dc493d8e95d237cbc18"
format_stamp: "Formatted at 2026-06-18 22:43:54 on dist-test-slave-lnj3"
server_key: "5e9ee6a319593f7266811f7318d1e3e1"
server_key_iv: "98c928160d7b26837c009c2db78379b2"
server_key_version: "encryptionkey@0"
I20260618 22:43:54.951236 27596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "7dff1db33ee04dc493d8e95d237cbc18"
format_stamp: "Formatted at 2026-06-18 22:43:54 on dist-test-slave-lnj3"
server_key: "5e9ee6a319593f7266811f7318d1e3e1"
server_key_iv: "98c928160d7b26837c009c2db78379b2"
server_key_version: "encryptionkey@0"
I20260618 22:43:54.956333 27596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260618 22:43:54.959692 27611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:54.960981 27596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:43:54.961133 27596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "7dff1db33ee04dc493d8e95d237cbc18"
format_stamp: "Formatted at 2026-06-18 22:43:54 on dist-test-slave-lnj3"
server_key: "5e9ee6a319593f7266811f7318d1e3e1"
server_key_iv: "98c928160d7b26837c009c2db78379b2"
server_key_version: "encryptionkey@0"
I20260618 22:43:54.961310 27596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:54.966907 27596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:54.971101 27596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:54.971386 27596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:54.988710 27596 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36227
I20260618 22:43:54.988745 27662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36227 every 8 connection(s)
I20260618 22:43:54.990065 27596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:43:54.993697 27663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:54.995572 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 27596
I20260618 22:43:54.995718 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:43:54.996132 5975 external_mini_cluster.cc:1428] Setting key 74b4cc89337315584cab355932fbc9cb
I20260618 22:43:55.005628 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:54.998104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:37985 (local address 127.0.0.1:36227)
0618 22:43:54.998974 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:54.998992 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:54.999050 (+ 58us) server_negotiation.cc:407] Connection header received
0618 22:43:55.001381 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:55.001420 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:55.001908 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:55.002387 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:55.003426 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:55.003435 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:55.003458 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:55.003476 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:55.003553 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:55.003589 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:55.003809 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:55.004010 (+ 201us) server_negotiation.cc:299] Negotiation successful
0618 22:43:55.004251 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":319,"threads_started":1}
I20260618 22:43:55.007397 27663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18: Bootstrap starting.
I20260618 22:43:55.009425 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:43:55.010887 27663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:55.012147 27663 log.cc:824] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:55.015543 27663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18: No bootstrap required, opened a new log
I20260618 22:43:55.017619 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:55.015305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:38207 (local address 127.0.0.1:36227)
0618 22:43:55.015474 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:55.015480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:55.015753 (+ 273us) server_negotiation.cc:407] Connection header received
0618 22:43:55.015923 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:55.015928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:55.016000 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:55.016115 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:55.016987 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:55.016993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:55.016998 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:55.017016 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:55.017059 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:55.017079 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:55.017193 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:55.017251 (+ 58us) server_negotiation.cc:299] Negotiation successful
0618 22:43:55.017318 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 22:43:55.024006 27663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dff1db33ee04dc493d8e95d237cbc18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36227 } }
I20260618 22:43:55.024372 27663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:55.024451 27663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dff1db33ee04dc493d8e95d237cbc18, State: Initialized, Role: FOLLOWER
I20260618 22:43:55.025007 27663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dff1db33ee04dc493d8e95d237cbc18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36227 } }
I20260618 22:43:55.025218 27663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:43:55.025319 27663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:43:55.025455 27663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:55.027638 27663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dff1db33ee04dc493d8e95d237cbc18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36227 } }
I20260618 22:43:55.028164 27663 leader_election.cc:302] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dff1db33ee04dc493d8e95d237cbc18; no voters:
I20260618 22:43:55.028789 27663 leader_election.cc:288] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:43:55.028920 27673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:55.029551 27673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18 [term 1 LEADER]: Becoming Leader. State: Replica: 7dff1db33ee04dc493d8e95d237cbc18, State: Running, Role: LEADER
I20260618 22:43:55.030078 27673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dff1db33ee04dc493d8e95d237cbc18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36227 } }
I20260618 22:43:55.030833 27663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:43:55.032725 27674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dff1db33ee04dc493d8e95d237cbc18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dff1db33ee04dc493d8e95d237cbc18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36227 } } }
I20260618 22:43:55.033156 27674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:55.033073 27675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dff1db33ee04dc493d8e95d237cbc18. Latest consensus state: current_term: 1 leader_uuid: "7dff1db33ee04dc493d8e95d237cbc18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dff1db33ee04dc493d8e95d237cbc18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36227 } } }
I20260618 22:43:55.033329 27675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:55.034520 27681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:43:55.039176 27681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:43:55.047899 27681 catalog_manager.cc:1347] Generated new cluster ID: c57b4a64b0174546ae0cc148ced7fa5f
I20260618 22:43:55.048040 27681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:43:55.071612 27681 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:43:55.072104 27681 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:43:55.086949 27681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dff1db33ee04dc493d8e95d237cbc18: Generated new TSK 0
I20260618 22:43:55.088063 27681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:43:55.096796 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 27596
W20260618 22:43:55.097946 27658 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.18:37985
W20260618 22:43:55.098599 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 27596: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:43:55.098764 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 27596
2026-06-18T22:43:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1769 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:43:55 dist-test-slave-lnj3 krb5kdc[27694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:43:55 dist-test-slave-lnj3 krb5kdc[27694](info): set up 2 sockets
Jun 18 22:43:55 dist-test-slave-lnj3 krb5kdc[27694](info): commencing operation
krb5kdc: starting...
W20260618 22:43:56.110782 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:43:56 dist-test-slave-lnj3 krb5kdc[27694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:43:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:43:56.345301 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:41281
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:41281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:56.667057 27707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:56.667358 27707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:56.667485 27707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:56.674642 27707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:43:56.674757 27707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:56.674815 27707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:56.674897 27707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:43:56.674955 27707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:43:56.684850 27707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:41281
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:41281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:56.686144 27707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:56.687512 27707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:56.698401 27715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:56.698439 27712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:56.698474 27713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:56.699486 27707 server_base.cc:1034] running on GCE node
I20260618 22:43:56.700186 27707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:56.701488 27707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:56.702736 27707 hybrid_clock.cc:648] HybridClock initialized: now 1781822636702654 us; error 46 us; skew 500 ppm
I20260618 22:43:56.703063 27707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:56 dist-test-slave-lnj3 krb5kdc[27694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822636, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:56.706007 27707 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:43:56.707410 27707 webserver.cc:460] Webserver started at http://127.5.213.254:33075/ using document root <none> and password file <none>
I20260618 22:43:56.708036 27707 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:56.708150 27707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:56.708444 27707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:56.711302 27707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "d4ce727b3cb74312bd6878ea91e94995"
format_stamp: "Formatted at 2026-06-18 22:43:56 on dist-test-slave-lnj3"
server_key: "eaa45c465626805a14bbd7851a866358"
server_key_iv: "98c2c14b744467e43fd710abcb3fff60"
server_key_version: "encryptionkey@0"
I20260618 22:43:56.712031 27707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "d4ce727b3cb74312bd6878ea91e94995"
format_stamp: "Formatted at 2026-06-18 22:43:56 on dist-test-slave-lnj3"
server_key: "eaa45c465626805a14bbd7851a866358"
server_key_iv: "98c2c14b744467e43fd710abcb3fff60"
server_key_version: "encryptionkey@0"
I20260618 22:43:56.716879 27707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:43:56.720157 27722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:56.721385 27707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260618 22:43:56.721539 27707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "d4ce727b3cb74312bd6878ea91e94995"
format_stamp: "Formatted at 2026-06-18 22:43:56 on dist-test-slave-lnj3"
server_key: "eaa45c465626805a14bbd7851a866358"
server_key_iv: "98c2c14b744467e43fd710abcb3fff60"
server_key_version: "encryptionkey@0"
I20260618 22:43:56.721719 27707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:56.754472 27707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:56.758457 27707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:56.758764 27707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:56.776124 27707 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:41281
I20260618 22:43:56.776172 27773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:41281 every 8 connection(s)
I20260618 22:43:56.777452 27707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:43:56.779663 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 27707
I20260618 22:43:56.779855 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:43:56.780267 5975 external_mini_cluster.cc:1428] Setting key c08e766c7c0caa703e91fdaf30ac4972
I20260618 22:43:56.781654 27774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 18 22:43:56 dist-test-slave-lnj3 krb5kdc[27694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:43:56.793689 27774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995: Bootstrap starting.
I20260618 22:43:56.797781 27774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:56.799074 27774 log.cc:824] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:56.802424 27774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995: No bootstrap required, opened a new log
I20260618 22:43:56.802665 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:56.782098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38561 (local address 127.5.213.254:41281)
0618 22:43:56.782893 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:56.782905 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:56.782944 (+ 39us) server_negotiation.cc:407] Connection header received
0618 22:43:56.784506 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:56.784528 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:56.784926 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:56.785296 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:56.786762 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:56.788078 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:56.789290 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:56.789951 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:56.790115 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:56.794325 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:56.794335 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:56.794355 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:56.794401 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:56.798174 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:56.799020 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:56.799029 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:56.799037 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:56.799289 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:56.799735 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:56.799741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:56.799745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:56.800303 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:56.800612 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:56.801008 (+ 396us) server_negotiation.cc:299] Negotiation successful
0618 22:43:56.801359 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":342,"threads_started":1}
I20260618 22:43:56.806452 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:43:56.810705 27774 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ce727b3cb74312bd6878ea91e94995" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 41281 } }
I20260618 22:43:56.811084 27774 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:56.811183 27774 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4ce727b3cb74312bd6878ea91e94995, State: Initialized, Role: FOLLOWER
I20260618 22:43:56.811792 27774 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ce727b3cb74312bd6878ea91e94995" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 41281 } }
I20260618 22:43:56.812006 27774 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:43:56.812132 27774 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:43:56.812276 27774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:56.815048 27774 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ce727b3cb74312bd6878ea91e94995" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 41281 } }
I20260618 22:43:56.815636 27774 leader_election.cc:302] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4ce727b3cb74312bd6878ea91e94995; no voters:
I20260618 22:43:56.816303 27774 leader_election.cc:288] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:43:56.816459 27784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:56.818496 27774 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:43:56.819463 27784 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995 [term 1 LEADER]: Becoming Leader. State: Replica: d4ce727b3cb74312bd6878ea91e94995, State: Running, Role: LEADER
I20260618 22:43:56.820035 27784 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ce727b3cb74312bd6878ea91e94995" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 41281 } }
I20260618 22:43:56.823264 27786 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4ce727b3cb74312bd6878ea91e94995. Latest consensus state: current_term: 1 leader_uuid: "d4ce727b3cb74312bd6878ea91e94995" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ce727b3cb74312bd6878ea91e94995" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 41281 } } }
I20260618 22:43:56.823850 27786 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:56.826387 27785 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4ce727b3cb74312bd6878ea91e94995" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ce727b3cb74312bd6878ea91e94995" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 41281 } } }
I20260618 22:43:56.827209 27785 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995 [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:56.827713 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:56.810484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33925 (local address 127.5.213.254:41281)
0618 22:43:56.810685 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:56.810694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:56.811051 (+ 357us) server_negotiation.cc:407] Connection header received
0618 22:43:56.811213 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:56.811219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:56.811299 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:56.811399 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:56.812896 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:56.814038 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:56.815245 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:56.815877 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:56.816003 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:56.819274 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:56.819280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:56.819285 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:43:56.819326 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:56.823151 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:56.824103 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:56.824109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:56.824111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:56.824377 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:43:56.824998 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:43:56.825002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:43:56.825005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:43:56.825374 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:56.826941 (+ 1567us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:56.827123 (+ 182us) server_negotiation.cc:299] Negotiation successful
0618 22:43:56.827319 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 22:43:56.834218 27792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:43:56.839875 27792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:43:56.848915 27792 catalog_manager.cc:1347] Generated new cluster ID: 149db63c69fa4775b1147ca797cb845c
I20260618 22:43:56.849038 27792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:43:56.865149 27792 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:43:56.866423 27792 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:43:56.878855 27792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4ce727b3cb74312bd6878ea91e94995: Generated new TSK 0
I20260618 22:43:56.879750 27792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:43:56.889071 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 27707
I20260618 22:43:56.890785 27769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:43:56.981389 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 27707
W20260618 22:43:56.995081 27692 connection.cc:535] client connection to 127.5.213.254:41281 recv error: Network error: failed to read from TLS socket (remote: 127.5.213.254:41281): Connection reset by peer (error 104)
2026-06-18T22:43:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1883 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:43:57 dist-test-slave-lnj3 krb5kdc[27806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:43:57 dist-test-slave-lnj3 krb5kdc[27806](info): set up 2 sockets
Jun 18 22:43:57 dist-test-slave-lnj3 krb5kdc[27806](info): commencing operation
krb5kdc: starting...
W20260618 22:43:57.998894 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:43:58 dist-test-slave-lnj3 krb5kdc[27806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:43:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:43:58.240252 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:35665
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:35665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:43:58.558815 27819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:43:58.559116 27819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:43:58.559249 27819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:43:58.566696 27819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:43:58.566818 27819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:43:58.566891 27819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:43:58.566973 27819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:43:58.567034 27819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:43:58.577040 27819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:35665
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:35665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:43:58.578328 27819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:43:58.579658 27819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:43:58.590786 27824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:58.590860 27825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:43:58.591076 27827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:43:58.591800 27819 server_base.cc:1034] running on GCE node
I20260618 22:43:58.592553 27819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:43:58.593926 27819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:43:58.595137 27819 hybrid_clock.cc:648] HybridClock initialized: now 1781822638595084 us; error 25 us; skew 500 ppm
I20260618 22:43:58.595463 27819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:43:58 dist-test-slave-lnj3 krb5kdc[27806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822638, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:43:58.598379 27819 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:43:58.599740 27819 webserver.cc:460] Webserver started at http://127.5.213.254:40427/ using document root <none> and password file <none>
I20260618 22:43:58.600348 27819 fs_manager.cc:359] Metadata directory not provided
I20260618 22:43:58.600440 27819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:43:58.600679 27819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:43:58.603561 27819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "1969e3e4a9984b429c7b5db3eacae5dc"
format_stamp: "Formatted at 2026-06-18 22:43:58 on dist-test-slave-lnj3"
server_key: "6edc460f70a0ae9e11669712fd7b2063"
server_key_iv: "45077c09871c8bee5acaa2c88412f007"
server_key_version: "encryptionkey@0"
I20260618 22:43:58.604285 27819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "1969e3e4a9984b429c7b5db3eacae5dc"
format_stamp: "Formatted at 2026-06-18 22:43:58 on dist-test-slave-lnj3"
server_key: "6edc460f70a0ae9e11669712fd7b2063"
server_key_iv: "45077c09871c8bee5acaa2c88412f007"
server_key_version: "encryptionkey@0"
I20260618 22:43:58.609222 27819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260618 22:43:58.612643 27834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:43:58.613993 27819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:43:58.614162 27819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "1969e3e4a9984b429c7b5db3eacae5dc"
format_stamp: "Formatted at 2026-06-18 22:43:58 on dist-test-slave-lnj3"
server_key: "6edc460f70a0ae9e11669712fd7b2063"
server_key_iv: "45077c09871c8bee5acaa2c88412f007"
server_key_version: "encryptionkey@0"
I20260618 22:43:58.614346 27819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:43:58.629644 27819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:43:58.633535 27819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:43:58.633783 27819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:43:58.650389 27819 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:35665
I20260618 22:43:58.650409 27885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:35665 every 8 connection(s)
I20260618 22:43:58.651679 27819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:43:58.652843 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 27819
I20260618 22:43:58.652997 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:43:58.653402 5975 external_mini_cluster.cc:1428] Setting key 44f66c255a8a84b43b4cbd38d7510a49
I20260618 22:43:58.655687 27886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:43:58.666491 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:58.654972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50169 (local address 127.5.213.254:35665)
0618 22:43:58.655703 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:58.655714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:58.655753 (+ 39us) server_negotiation.cc:407] Connection header received
0618 22:43:58.657457 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:58.657480 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:58.657869 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:58.658194 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:58.658771 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:58.660070 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:58.661869 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:58.662504 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:58.662703 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:58.664020 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:58.664029 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:58.664043 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:58.664695 (+ 652us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:58.664799 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:58.664828 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:58.664970 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:58.665278 (+ 308us) server_negotiation.cc:299] Negotiation successful
0618 22:43:58.665468 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":327,"threads_started":1}
I20260618 22:43:58.667424 27886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc: Bootstrap starting.
I20260618 22:43:58.670320 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:43:58.670603 27886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc: Neither blocks nor log segments found. Creating new log.
I20260618 22:43:58.671554 27886 log.cc:824] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc: Log is configured to *not* fsync() on all Append() calls
I20260618 22:43:58.674388 27886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc: No bootstrap required, opened a new log
I20260618 22:43:58.681339 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:43:58.674709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40737 (local address 127.5.213.254:35665)
0618 22:43:58.674904 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0618 22:43:58.674911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:43:58.675385 (+ 474us) server_negotiation.cc:407] Connection header received
0618 22:43:58.675524 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:43:58.675529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:43:58.675599 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:43:58.675696 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:43:58.676137 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:58.677317 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:58.678602 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:43:58.679332 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:43:58.679458 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:43:58.680527 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:43:58.680534 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:43:58.680539 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:43:58.680559 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:43:58.680603 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:43:58.680623 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:43:58.680737 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0618 22:43:58.680928 (+ 191us) server_negotiation.cc:299] Negotiation successful
0618 22:43:58.680977 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:43:58.683314 27886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1969e3e4a9984b429c7b5db3eacae5dc" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 35665 } }
I20260618 22:43:58.683669 27886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:43:58.683750 27886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1969e3e4a9984b429c7b5db3eacae5dc, State: Initialized, Role: FOLLOWER
I20260618 22:43:58.684243 27886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1969e3e4a9984b429c7b5db3eacae5dc" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 35665 } }
I20260618 22:43:58.684432 27886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:43:58.684527 27886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:43:58.684646 27886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:43:58.686757 27886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1969e3e4a9984b429c7b5db3eacae5dc" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 35665 } }
I20260618 22:43:58.687208 27886 leader_election.cc:302] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1969e3e4a9984b429c7b5db3eacae5dc; no voters:
I20260618 22:43:58.687762 27886 leader_election.cc:288] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:43:58.687944 27896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:43:58.688868 27896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc [term 1 LEADER]: Becoming Leader. State: Replica: 1969e3e4a9984b429c7b5db3eacae5dc, State: Running, Role: LEADER
I20260618 22:43:58.689532 27896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1969e3e4a9984b429c7b5db3eacae5dc" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 35665 } }
I20260618 22:43:58.689978 27886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:43:58.692085 27898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1969e3e4a9984b429c7b5db3eacae5dc. Latest consensus state: current_term: 1 leader_uuid: "1969e3e4a9984b429c7b5db3eacae5dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1969e3e4a9984b429c7b5db3eacae5dc" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 35665 } } }
I20260618 22:43:58.692441 27898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:58.692934 27897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1969e3e4a9984b429c7b5db3eacae5dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1969e3e4a9984b429c7b5db3eacae5dc" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 35665 } } }
I20260618 22:43:58.693222 27897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc [sys.catalog]: This master's current role is: LEADER
I20260618 22:43:58.694383 27902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:43:58.698653 27902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:43:58.707391 27902 catalog_manager.cc:1347] Generated new cluster ID: e637079a9c8c425a97f43814a18550e9
I20260618 22:43:58.707494 27902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:43:58.724200 27902 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:43:58.725459 27902 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:43:58.737821 27902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1969e3e4a9984b429c7b5db3eacae5dc: Generated new TSK 0
I20260618 22:43:58.738735 27902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:43:58.748529 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 27819
W20260618 22:43:58.749624 27881 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50169
W20260618 22:43:58.750419 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 27819: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:43:58.750555 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 27819
W20260618 22:43:58.763806 27804 connection.cc:535] client connection to 127.5.213.254:35665 recv error: Network error: failed to read from TLS socket (remote: 127.5.213.254:35665): Connection reset by peer (error 104)
2026-06-18T22:43:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1768 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:43:58 dist-test-slave-lnj3 krb5kdc[27918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:43:58 dist-test-slave-lnj3 krb5kdc[27918](info): set up 2 sockets
Jun 18 22:43:58 dist-test-slave-lnj3 krb5kdc[27918](info): commencing operation
krb5kdc: starting...
W20260618 22:43:59.754997 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:43:59 dist-test-slave-lnj3 krb5kdc[27918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:43:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:43:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:43:59.991876 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:35361
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:35981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:35361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:44:00.305392 27931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:44:00.305692 27931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:44:00.305833 27931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:44:00.312964 27931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:44:00.313084 27931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:44:00.313160 27931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:44:00.313236 27931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:44:00.313295 27931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:44:00.323194 27931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:35981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:35361
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:35361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:44:00.324466 27931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:44:00.325750 27931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:44:00.336820 27939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:44:00.336891 27937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:44:00.336865 27936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:44:00.337703 27931 server_base.cc:1034] running on GCE node
I20260618 22:44:00.338429 27931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:44:00.339886 27931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:44:00.341101 27931 hybrid_clock.cc:648] HybridClock initialized: now 1781822640341041 us; error 47 us; skew 500 ppm
I20260618 22:44:00.341454 27931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:44:00 dist-test-slave-lnj3 krb5kdc[27918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822640, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:44:00.348806 27931 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:44:00.350195 27931 webserver.cc:460] Webserver started at http://127.5.213.254:33945/ using document root <none> and password file <none>
I20260618 22:44:00.350898 27931 fs_manager.cc:359] Metadata directory not provided
I20260618 22:44:00.351009 27931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:44:00.351295 27931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:44:00.354239 27931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "643be6813b73416ba26acbded4256b28"
format_stamp: "Formatted at 2026-06-18 22:44:00 on dist-test-slave-lnj3"
server_key: "40300fd0b37635fa552031b55fd3463b"
server_key_iv: "420fc13922c2b27c175c94e2a540826a"
server_key_version: "encryptionkey@0"
I20260618 22:44:00.355012 27931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "643be6813b73416ba26acbded4256b28"
format_stamp: "Formatted at 2026-06-18 22:44:00 on dist-test-slave-lnj3"
server_key: "40300fd0b37635fa552031b55fd3463b"
server_key_iv: "420fc13922c2b27c175c94e2a540826a"
server_key_version: "encryptionkey@0"
I20260618 22:44:00.360000 27931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260618 22:44:00.363364 27946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:44:00.364671 27931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260618 22:44:00.364830 27931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "643be6813b73416ba26acbded4256b28"
format_stamp: "Formatted at 2026-06-18 22:44:00 on dist-test-slave-lnj3"
server_key: "40300fd0b37635fa552031b55fd3463b"
server_key_iv: "420fc13922c2b27c175c94e2a540826a"
server_key_version: "encryptionkey@0"
I20260618 22:44:00.365016 27931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:44:00.370605 27931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:44:00.374496 27931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:44:00.374806 27931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:44:00.391746 27931 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:35361
I20260618 22:44:00.391780 27997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:35361 every 8 connection(s)
I20260618 22:44:00.393088 27931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:44:00.393862 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 27931
I20260618 22:44:00.394042 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:44:00.394450 5975 external_mini_cluster.cc:1428] Setting key 6a1a25fa995c1fd07f0a1b9f75f96c11
I20260618 22:44:00.397389 27998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:44:00.403103 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:00.396247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56797 (local address 127.5.213.254:35361)
0618 22:44:00.397023 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:00.397039 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:00.397088 (+ 49us) server_negotiation.cc:407] Connection header received
0618 22:44:00.399335 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:00.399361 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:00.399767 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:00.400208 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:00.401167 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:00.401175 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:00.401191 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:44:00.401208 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:00.401300 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:44:00.401327 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:00.401537 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:00.401665 (+ 128us) server_negotiation.cc:299] Negotiation successful
0618 22:44:00.401849 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":297,"threads_started":1}
I20260618 22:44:00.406487 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:44:00.410589 27998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28: Bootstrap starting.
I20260618 22:44:00.413960 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:00.411404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52715 (local address 127.5.213.254:35361)
0618 22:44:00.411587 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:00.411593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:00.412095 (+ 502us) server_negotiation.cc:407] Connection header received
0618 22:44:00.412278 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:00.412282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:00.412337 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:00.412414 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:00.413355 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:00.413359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:00.413362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:44:00.413376 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:00.413415 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:44:00.413430 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:00.413513 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:00.413651 (+ 138us) server_negotiation.cc:299] Negotiation successful
0618 22:44:00.413695 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:44:00.414438 27998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28: Neither blocks nor log segments found. Creating new log.
I20260618 22:44:00.415580 27998 log.cc:824] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28: Log is configured to *not* fsync() on all Append() calls
I20260618 22:44:00.418322 27998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28: No bootstrap required, opened a new log
I20260618 22:44:00.425451 27998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643be6813b73416ba26acbded4256b28" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 35361 } }
I20260618 22:44:00.425808 27998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:44:00.425899 27998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 643be6813b73416ba26acbded4256b28, State: Initialized, Role: FOLLOWER
I20260618 22:44:00.426420 27998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643be6813b73416ba26acbded4256b28" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 35361 } }
I20260618 22:44:00.426646 27998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:44:00.426795 27998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:44:00.426978 27998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:44:00.429080 27998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643be6813b73416ba26acbded4256b28" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 35361 } }
I20260618 22:44:00.429530 27998 leader_election.cc:302] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 643be6813b73416ba26acbded4256b28; no voters:
I20260618 22:44:00.430037 27998 leader_election.cc:288] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:44:00.430145 28008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:44:00.430763 28008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28 [term 1 LEADER]: Becoming Leader. State: Replica: 643be6813b73416ba26acbded4256b28, State: Running, Role: LEADER
I20260618 22:44:00.431267 28008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643be6813b73416ba26acbded4256b28" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 35361 } }
I20260618 22:44:00.431942 27998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:44:00.435334 28010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 643be6813b73416ba26acbded4256b28. Latest consensus state: current_term: 1 leader_uuid: "643be6813b73416ba26acbded4256b28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643be6813b73416ba26acbded4256b28" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 35361 } } }
I20260618 22:44:00.435719 28010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28 [sys.catalog]: This master's current role is: LEADER
I20260618 22:44:00.435314 28009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "643be6813b73416ba26acbded4256b28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643be6813b73416ba26acbded4256b28" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 35361 } } }
I20260618 22:44:00.436515 28015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:44:00.436491 28009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28 [sys.catalog]: This master's current role is: LEADER
I20260618 22:44:00.441238 28015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:44:00.451251 28015 catalog_manager.cc:1347] Generated new cluster ID: cf53dec77a6b4ee79277fc6c72501979
I20260618 22:44:00.451377 28015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:44:00.470774 28015 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:44:00.471262 28015 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:44:00.487789 28015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 643be6813b73416ba26acbded4256b28: Generated new TSK 0
I20260618 22:44:00.488991 28015 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:44:00.502786 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 27931
W20260618 22:44:00.504199 27993 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56797
W20260618 22:44:00.504963 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 27931: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:44:00.505110 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 27931
2026-06-18T22:44:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1754 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:44:00 dist-test-slave-lnj3 krb5kdc[28029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:44:00 dist-test-slave-lnj3 krb5kdc[28029](info): set up 2 sockets
Jun 18 22:44:00 dist-test-slave-lnj3 krb5kdc[28029](info): commencing operation
krb5kdc: starting...
W20260618 22:44:01.518922 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:44:01 dist-test-slave-lnj3 krb5kdc[28029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:44:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:44:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:44:01.761606 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:42533
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:42533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:44:02.087105 28042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:44:02.087397 28042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:44:02.087533 28042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:44:02.094813 28042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:44:02.094944 28042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:44:02.095022 28042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:44:02.095110 28042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:44:02.095168 28042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:44:02.105079 28042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:42533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:42533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:44:02.106317 28042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:44:02.107683 28042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:44:02.119172 28048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:44:02.119207 28050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:44:02.119122 28047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:44:02.120388 28042 server_base.cc:1034] running on GCE node
I20260618 22:44:02.121186 28042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:44:02.122587 28042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:44:02.123916 28042 hybrid_clock.cc:648] HybridClock initialized: now 1781822642123858 us; error 52 us; skew 500 ppm
I20260618 22:44:02.124264 28042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:44:02 dist-test-slave-lnj3 krb5kdc[28029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822642, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:44:02.127296 28042 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:44:02.128763 28042 webserver.cc:460] Webserver started at http://127.5.213.254:46603/ using document root <none> and password file <none>
I20260618 22:44:02.129426 28042 fs_manager.cc:359] Metadata directory not provided
I20260618 22:44:02.129539 28042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:44:02.129825 28042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:44:02.132824 28042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "42633ba36df4499ca8783b852dbeefd7"
format_stamp: "Formatted at 2026-06-18 22:44:02 on dist-test-slave-lnj3"
server_key: "4592be9133ce618d85c21f06baa602b0"
server_key_iv: "b4a324f7592960f1e422e4f616ce40c6"
server_key_version: "encryptionkey@0"
I20260618 22:44:02.133564 28042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "42633ba36df4499ca8783b852dbeefd7"
format_stamp: "Formatted at 2026-06-18 22:44:02 on dist-test-slave-lnj3"
server_key: "4592be9133ce618d85c21f06baa602b0"
server_key_iv: "b4a324f7592960f1e422e4f616ce40c6"
server_key_version: "encryptionkey@0"
I20260618 22:44:02.138727 28042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260618 22:44:02.142148 28057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:44:02.143508 28042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:44:02.143669 28042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "42633ba36df4499ca8783b852dbeefd7"
format_stamp: "Formatted at 2026-06-18 22:44:02 on dist-test-slave-lnj3"
server_key: "4592be9133ce618d85c21f06baa602b0"
server_key_iv: "b4a324f7592960f1e422e4f616ce40c6"
server_key_version: "encryptionkey@0"
I20260618 22:44:02.143852 28042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:44:02.154012 28042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:44:02.157943 28042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:44:02.158185 28042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:44:02.175009 28042 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:42533
I20260618 22:44:02.175031 28108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:42533 every 8 connection(s)
I20260618 22:44:02.176421 28042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:44:02.180382 28109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:44:02.185036 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 28042
I20260618 22:44:02.185225 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:44:02.185739 5975 external_mini_cluster.cc:1428] Setting key 6fb894bb19e44ba7afe8352c908c289a
I20260618 22:44:02.191314 28109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7: Bootstrap starting.
I20260618 22:44:02.194244 28109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7: Neither blocks nor log segments found. Creating new log.
I20260618 22:44:02.195251 28109 log.cc:824] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7: Log is configured to *not* fsync() on all Append() calls
I20260618 22:44:02.197831 28109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7: No bootstrap required, opened a new log
Jun 18 22:44:02 dist-test-slave-lnj3 krb5kdc[28029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:44:02.207767 28112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:02.187432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53773 (local address 127.5.213.254:42533)
0618 22:44:02.188168 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:02.188179 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:02.188218 (+ 39us) server_negotiation.cc:407] Connection header received
0618 22:44:02.190044 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:02.190067 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:02.190446 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:02.190819 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:02.192212 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:02.193500 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:02.194760 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:02.195425 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:02.195586 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:02.199424 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:02.199434 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:02.199451 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:02.199494 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:02.203366 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:02.204033 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:02.204042 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:02.204051 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:02.204254 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:02.204575 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:02.204580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:02.204583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:02.205258 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:02.205553 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:02.205877 (+ 324us) server_negotiation.cc:299] Negotiation successful
0618 22:44:02.206404 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":327,"threads_started":1}
I20260618 22:44:02.208202 28109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42633ba36df4499ca8783b852dbeefd7" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42533 } }
I20260618 22:44:02.208643 28109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:44:02.208752 28109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42633ba36df4499ca8783b852dbeefd7, State: Initialized, Role: FOLLOWER
I20260618 22:44:02.209339 28109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42633ba36df4499ca8783b852dbeefd7" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42533 } }
I20260618 22:44:02.209555 28109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:44:02.209692 28109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:44:02.209847 28109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:44:02.211535 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:44:02.212445 28109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42633ba36df4499ca8783b852dbeefd7" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42533 } }
I20260618 22:44:02.212929 28109 leader_election.cc:302] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42633ba36df4499ca8783b852dbeefd7; no voters:
I20260618 22:44:02.213493 28109 leader_election.cc:288] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:44:02.213593 28114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:44:02.219475 28114 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7 [term 1 LEADER]: Becoming Leader. State: Replica: 42633ba36df4499ca8783b852dbeefd7, State: Running, Role: LEADER
I20260618 22:44:02.220098 28114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42633ba36df4499ca8783b852dbeefd7" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42533 } }
I20260618 22:44:02.220675 28109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:44:02.223379 28116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42633ba36df4499ca8783b852dbeefd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42633ba36df4499ca8783b852dbeefd7" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42533 } } }
I20260618 22:44:02.223953 28117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42633ba36df4499ca8783b852dbeefd7. Latest consensus state: current_term: 1 leader_uuid: "42633ba36df4499ca8783b852dbeefd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42633ba36df4499ca8783b852dbeefd7" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 42533 } } }
I20260618 22:44:02.224640 28116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7 [sys.catalog]: This master's current role is: LEADER
I20260618 22:44:02.224684 28117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7 [sys.catalog]: This master's current role is: LEADER
I20260618 22:44:02.229049 28126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:44:02.234609 28112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:02.215694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55365 (local address 127.5.213.254:42533)
0618 22:44:02.215894 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:02.215900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:02.220185 (+ 4285us) server_negotiation.cc:407] Connection header received
0618 22:44:02.220363 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:02.220369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:02.220442 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:02.220529 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:02.222003 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:02.223091 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:02.224560 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:02.225246 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:02.225374 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:02.228538 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:02.228545 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:02.228551 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:02.228597 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:02.231880 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:02.232592 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:02.232596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:02.232598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:02.232723 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:02.233053 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:02.233058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:02.233061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:02.233359 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:02.233578 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:02.233893 (+ 315us) server_negotiation.cc:299] Negotiation successful
0618 22:44:02.234100 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260618 22:44:02.236449 28126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:44:02.245934 28126 catalog_manager.cc:1347] Generated new cluster ID: f31a91c3f04942f080636a875818029a
I20260618 22:44:02.246048 28126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:44:02.261317 28126 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:44:02.263145 28126 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:44:02.270388 28126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42633ba36df4499ca8783b852dbeefd7: Generated new TSK 0
I20260618 22:44:02.271449 28126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:44:02.278080 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 28042
I20260618 22:44:02.279084 28104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:44:02.367690 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 28042
2026-06-18T22:44:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1862 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:44:02 dist-test-slave-lnj3 krb5kdc[28141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:44:02 dist-test-slave-lnj3 krb5kdc[28141](info): set up 2 sockets
Jun 18 22:44:02 dist-test-slave-lnj3 krb5kdc[28141](info): commencing operation
krb5kdc: starting...
W20260618 22:44:03.379004 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:44:03 dist-test-slave-lnj3 krb5kdc[28141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:44:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:44:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:44:03.624059 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:40189
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:40189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:44:03.938598 28154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:44:03.938912 28154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:44:03.939041 28154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:44:03.946164 28154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:44:03.946269 28154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:44:03.946314 28154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:44:03.946377 28154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:44:03.946440 28154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:44:03.956126 28154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:40189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:40189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:44:03.957350 28154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:44:03.958600 28154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:44:03.969813 28162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:44:03.969862 28159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:44:03.969838 28160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:44:03.970942 28154 server_base.cc:1034] running on GCE node
I20260618 22:44:03.971692 28154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:44:03.973088 28154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:44:03.974308 28154 hybrid_clock.cc:648] HybridClock initialized: now 1781822643974256 us; error 42 us; skew 500 ppm
I20260618 22:44:03.974689 28154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:44:03 dist-test-slave-lnj3 krb5kdc[28141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822643, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:44:03.977792 28154 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:44:03.979265 28154 webserver.cc:460] Webserver started at http://127.5.213.254:38929/ using document root <none> and password file <none>
I20260618 22:44:03.979930 28154 fs_manager.cc:359] Metadata directory not provided
I20260618 22:44:03.980041 28154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:44:03.980310 28154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:44:03.983179 28154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "988ebdaa5b0a44f4b9299512334c028d"
format_stamp: "Formatted at 2026-06-18 22:44:03 on dist-test-slave-lnj3"
server_key: "689a1d8ce1f031bfba1ddd0538fc3a04"
server_key_iv: "5d00974e00bdb503df53fd4d000e71e3"
server_key_version: "encryptionkey@0"
I20260618 22:44:03.984102 28154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "988ebdaa5b0a44f4b9299512334c028d"
format_stamp: "Formatted at 2026-06-18 22:44:03 on dist-test-slave-lnj3"
server_key: "689a1d8ce1f031bfba1ddd0538fc3a04"
server_key_iv: "5d00974e00bdb503df53fd4d000e71e3"
server_key_version: "encryptionkey@0"
I20260618 22:44:03.990029 28154 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260618 22:44:03.993415 28169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:44:03.994764 28154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260618 22:44:03.994907 28154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "988ebdaa5b0a44f4b9299512334c028d"
format_stamp: "Formatted at 2026-06-18 22:44:03 on dist-test-slave-lnj3"
server_key: "689a1d8ce1f031bfba1ddd0538fc3a04"
server_key_iv: "5d00974e00bdb503df53fd4d000e71e3"
server_key_version: "encryptionkey@0"
I20260618 22:44:03.995076 28154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:44:04.029701 28154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:44:04.033596 28154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:44:04.033856 28154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:44:04.051870 28154 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:40189
I20260618 22:44:04.051888 28220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:40189 every 8 connection(s)
I20260618 22:44:04.053289 28154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:44:04.056936 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 28154
I20260618 22:44:04.057123 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:44:04.057269 28221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:44:04.057649 5975 external_mini_cluster.cc:1428] Setting key 42b037a6cbda1b959037f72f12d6102e
I20260618 22:44:04.070302 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:04.059360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60505 (local address 127.5.213.254:40189)
0618 22:44:04.060123 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:04.060134 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:04.060176 (+ 42us) server_negotiation.cc:407] Connection header received
0618 22:44:04.061887 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:04.061907 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:04.062280 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:04.062582 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:04.063142 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:04.064437 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:04.066137 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:04.066857 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:04.066993 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:04.068178 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:04.068185 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:04.068198 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:44:04.068722 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:04.068797 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:44:04.068828 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:04.068907 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:04.069110 (+ 203us) server_negotiation.cc:299] Negotiation successful
0618 22:44:04.069346 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":325,"threads_started":1}
I20260618 22:44:04.070567 28221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d: Bootstrap starting.
I20260618 22:44:04.073628 28221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d: Neither blocks nor log segments found. Creating new log.
I20260618 22:44:04.073786 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:44:04.074807 28221 log.cc:824] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d: Log is configured to *not* fsync() on all Append() calls
I20260618 22:44:04.077968 28221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d: No bootstrap required, opened a new log
I20260618 22:44:04.084967 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:04.078606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42139 (local address 127.5.213.254:40189)
0618 22:44:04.078803 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:04.078808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:04.079130 (+ 322us) server_negotiation.cc:407] Connection header received
0618 22:44:04.079390 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:04.079394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:04.079455 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:04.079543 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:04.080143 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:04.080938 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:04.082761 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:04.083239 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:04.083333 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:04.084341 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:04.084349 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:04.084352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:44:04.084369 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:04.084408 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:44:04.084423 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:04.084491 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:04.084618 (+ 127us) server_negotiation.cc:299] Negotiation successful
0618 22:44:04.084686 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260618 22:44:04.086962 28221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988ebdaa5b0a44f4b9299512334c028d" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40189 } }
I20260618 22:44:04.087298 28221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:44:04.087383 28221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 988ebdaa5b0a44f4b9299512334c028d, State: Initialized, Role: FOLLOWER
I20260618 22:44:04.087889 28221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988ebdaa5b0a44f4b9299512334c028d" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40189 } }
I20260618 22:44:04.088099 28221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:44:04.088217 28221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:44:04.088353 28221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:44:04.090469 28221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988ebdaa5b0a44f4b9299512334c028d" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40189 } }
I20260618 22:44:04.090955 28221 leader_election.cc:302] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 988ebdaa5b0a44f4b9299512334c028d; no voters:
I20260618 22:44:04.091480 28221 leader_election.cc:288] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:44:04.091632 28231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:44:04.092545 28231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d [term 1 LEADER]: Becoming Leader. State: Replica: 988ebdaa5b0a44f4b9299512334c028d, State: Running, Role: LEADER
I20260618 22:44:04.093037 28231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988ebdaa5b0a44f4b9299512334c028d" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40189 } }
I20260618 22:44:04.093487 28221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:44:04.095731 28232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "988ebdaa5b0a44f4b9299512334c028d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988ebdaa5b0a44f4b9299512334c028d" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40189 } } }
I20260618 22:44:04.095652 28233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 988ebdaa5b0a44f4b9299512334c028d. Latest consensus state: current_term: 1 leader_uuid: "988ebdaa5b0a44f4b9299512334c028d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988ebdaa5b0a44f4b9299512334c028d" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40189 } } }
I20260618 22:44:04.096060 28233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d [sys.catalog]: This master's current role is: LEADER
I20260618 22:44:04.096057 28232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d [sys.catalog]: This master's current role is: LEADER
I20260618 22:44:04.097802 28239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:44:04.102939 28239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:44:04.111969 28239 catalog_manager.cc:1347] Generated new cluster ID: ad5c0447432049f79ac0f6ca88510c8b
I20260618 22:44:04.112062 28239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:44:04.127964 28239 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:44:04.129300 28239 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:44:04.137961 28239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 988ebdaa5b0a44f4b9299512334c028d: Generated new TSK 0
I20260618 22:44:04.138865 28239 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:44:04.145282 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 28154
W20260618 22:44:04.146292 28216 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60505
W20260618 22:44:04.146936 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 28154: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:44:04.147084 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 28154
2026-06-18T22:44:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1778 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:44:04 dist-test-slave-lnj3 krb5kdc[28252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:44:04 dist-test-slave-lnj3 krb5kdc[28252](info): set up 2 sockets
Jun 18 22:44:04 dist-test-slave-lnj3 krb5kdc[28252](info): commencing operation
krb5kdc: starting...
W20260618 22:44:05.146924 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:44:05 dist-test-slave-lnj3 krb5kdc[28252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:44:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:44:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:44:05.383752 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:35329
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:35329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:44:05.707011 28265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:44:05.707283 28265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:44:05.707408 28265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:44:05.714555 28265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:44:05.714717 28265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:44:05.714795 28265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:44:05.714866 28265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:44:05.714922 28265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:44:05.724874 28265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:35329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:35329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:44:05.726218 28265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:44:05.727509 28265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:44:05.738833 28273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:44:05.738795 28270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:44:05.738857 28271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:44:05.739566 28265 server_base.cc:1034] running on GCE node
I20260618 22:44:05.740345 28265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:44:05.741732 28265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:44:05.742934 28265 hybrid_clock.cc:648] HybridClock initialized: now 1781822645742891 us; error 30 us; skew 500 ppm
I20260618 22:44:05.743270 28265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:44:05 dist-test-slave-lnj3 krb5kdc[28252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822645, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:44:05.751034 28265 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:44:05.752502 28265 webserver.cc:460] Webserver started at http://127.5.213.254:36351/ using document root <none> and password file <none>
I20260618 22:44:05.753177 28265 fs_manager.cc:359] Metadata directory not provided
I20260618 22:44:05.753293 28265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:44:05.753571 28265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:44:05.756541 28265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "2c11fc94054641f7bce5a04acd0ac2f5"
format_stamp: "Formatted at 2026-06-18 22:44:05 on dist-test-slave-lnj3"
server_key: "0f72b925f137b06dc353fbed7b3e790c"
server_key_iv: "d1e5439a3f65530f31945743ba66e5c1"
server_key_version: "encryptionkey@0"
I20260618 22:44:05.757287 28265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "2c11fc94054641f7bce5a04acd0ac2f5"
format_stamp: "Formatted at 2026-06-18 22:44:05 on dist-test-slave-lnj3"
server_key: "0f72b925f137b06dc353fbed7b3e790c"
server_key_iv: "d1e5439a3f65530f31945743ba66e5c1"
server_key_version: "encryptionkey@0"
I20260618 22:44:05.762310 28265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260618 22:44:05.765779 28280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:44:05.767201 28265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:44:05.767362 28265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "2c11fc94054641f7bce5a04acd0ac2f5"
format_stamp: "Formatted at 2026-06-18 22:44:05 on dist-test-slave-lnj3"
server_key: "0f72b925f137b06dc353fbed7b3e790c"
server_key_iv: "d1e5439a3f65530f31945743ba66e5c1"
server_key_version: "encryptionkey@0"
I20260618 22:44:05.767539 28265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:44:05.773022 28265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:44:05.777199 28265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:44:05.777457 28265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:44:05.794088 28265 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:35329
I20260618 22:44:05.794099 28331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:35329 every 8 connection(s)
I20260618 22:44:05.795522 28265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:44:05.797245 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 28265
I20260618 22:44:05.797430 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:44:05.797959 5975 external_mini_cluster.cc:1428] Setting key 2558930fdb1d9a47e979d1c751145326
I20260618 22:44:05.799973 28332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:44:05.805581 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:05.799719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34687 (local address 127.5.213.254:35329)
0618 22:44:05.800477 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:05.800489 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:05.800528 (+ 39us) server_negotiation.cc:407] Connection header received
0618 22:44:05.802099 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:05.802122 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:05.802457 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:05.802820 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:05.803850 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:05.803856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:05.803871 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:44:05.803905 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:05.803995 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:44:05.804026 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:05.804293 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:05.804427 (+ 134us) server_negotiation.cc:299] Negotiation successful
0618 22:44:05.804602 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":350,"threads_started":1}
I20260618 22:44:05.809182 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:44:05.811262 28332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5: Bootstrap starting.
I20260618 22:44:05.814677 28332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5: Neither blocks nor log segments found. Creating new log.
I20260618 22:44:05.815754 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:05.813342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49983 (local address 127.5.213.254:35329)
0618 22:44:05.813532 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:05.813538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:05.813870 (+ 332us) server_negotiation.cc:407] Connection header received
0618 22:44:05.814043 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:05.814046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:05.814095 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:05.814168 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:05.815129 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:05.815135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:05.815140 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:44:05.815158 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:05.815206 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:44:05.815225 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:05.815337 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:05.815417 (+ 80us) server_negotiation.cc:299] Negotiation successful
0618 22:44:05.815470 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260618 22:44:05.815930 28332 log.cc:824] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5: Log is configured to *not* fsync() on all Append() calls
I20260618 22:44:05.819046 28332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5: No bootstrap required, opened a new log
I20260618 22:44:05.826732 28332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c11fc94054641f7bce5a04acd0ac2f5" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 35329 } }
I20260618 22:44:05.827106 28332 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:44:05.827204 28332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c11fc94054641f7bce5a04acd0ac2f5, State: Initialized, Role: FOLLOWER
I20260618 22:44:05.827790 28332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c11fc94054641f7bce5a04acd0ac2f5" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 35329 } }
I20260618 22:44:05.828065 28332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:44:05.828207 28332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:44:05.828387 28332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:44:05.830507 28332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c11fc94054641f7bce5a04acd0ac2f5" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 35329 } }
I20260618 22:44:05.831009 28332 leader_election.cc:302] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c11fc94054641f7bce5a04acd0ac2f5; no voters:
I20260618 22:44:05.831562 28332 leader_election.cc:288] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:44:05.831694 28342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:44:05.832264 28342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5 [term 1 LEADER]: Becoming Leader. State: Replica: 2c11fc94054641f7bce5a04acd0ac2f5, State: Running, Role: LEADER
I20260618 22:44:05.832805 28342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c11fc94054641f7bce5a04acd0ac2f5" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 35329 } }
I20260618 22:44:05.833423 28332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:44:05.835669 28343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c11fc94054641f7bce5a04acd0ac2f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c11fc94054641f7bce5a04acd0ac2f5" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 35329 } } }
I20260618 22:44:05.835732 28344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c11fc94054641f7bce5a04acd0ac2f5. Latest consensus state: current_term: 1 leader_uuid: "2c11fc94054641f7bce5a04acd0ac2f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c11fc94054641f7bce5a04acd0ac2f5" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 35329 } } }
I20260618 22:44:05.836113 28344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5 [sys.catalog]: This master's current role is: LEADER
I20260618 22:44:05.836195 28343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5 [sys.catalog]: This master's current role is: LEADER
I20260618 22:44:05.837545 28348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:44:05.841420 28348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:44:05.851445 28348 catalog_manager.cc:1347] Generated new cluster ID: e6c5b967fabc498bab9613b26148f969
I20260618 22:44:05.851552 28348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:44:05.874095 28348 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:44:05.874491 28348 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:44:05.884812 28348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c11fc94054641f7bce5a04acd0ac2f5: Generated new TSK 0
I20260618 22:44:05.885589 28348 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:44:05.894999 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 28265
W20260618 22:44:05.896492 28327 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34687
W20260618 22:44:05.897183 5975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 28265: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260618 22:44:05.897316 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 28265
2026-06-18T22:44:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1750 ms)
[----------] 18 tests from AuthTokenIssuingTest (32588 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:44:05 dist-test-slave-lnj3 krb5kdc[28363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:44:05 dist-test-slave-lnj3 krb5kdc[28363](info): set up 2 sockets
Jun 18 22:44:05 dist-test-slave-lnj3 krb5kdc[28363](info): commencing operation
krb5kdc: starting...
W20260618 22:44:06.919008 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:44:07 dist-test-slave-lnj3 krb5kdc[28363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:44:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:44:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:44:07.164430 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:40785
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:40785,127.5.213.253:39699,127.5.213.252:41315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:44:07.484576 28376 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 22:44:07.484912 28376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:44:07.485011 28376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:44:07.485112 28376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:44:07.492517 28376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:44:07.492624 28376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:44:07.492694 28376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:44:07.492763 28376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:44:07.492846 28376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:44:07.502902 28376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:40785,127.5.213.253:39699,127.5.213.252:41315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:40785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:44:07.504181 28376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:44:07.505484 28376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:44:07.516623 28381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:44:07.516587 28382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:44:07.516705 28384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:44:07.517403 28376 server_base.cc:1034] running on GCE node
I20260618 22:44:07.518126 28376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:44:07.519507 28376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:44:07.520736 28376 hybrid_clock.cc:648] HybridClock initialized: now 1781822647520678 us; error 37 us; skew 500 ppm
I20260618 22:44:07.521032 28376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:44:07 dist-test-slave-lnj3 krb5kdc[28363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822647, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:44:07.524066 28376 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:44:07.525673 28376 webserver.cc:460] Webserver started at http://127.5.213.254:33063/ using document root <none> and password file <none>
I20260618 22:44:07.526382 28376 fs_manager.cc:359] Metadata directory not provided
I20260618 22:44:07.526492 28376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:44:07.526872 28376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:44:07.529995 28376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "f61e8a5bbddb4cb78ec5970b500b75b1"
format_stamp: "Formatted at 2026-06-18 22:44:07 on dist-test-slave-lnj3"
server_key: "cf0e12351435cc3033292b13451d8c50"
server_key_iv: "fb612a3304919705cae001fdac940d70"
server_key_version: "encryptionkey@0"
I20260618 22:44:07.530874 28376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "f61e8a5bbddb4cb78ec5970b500b75b1"
format_stamp: "Formatted at 2026-06-18 22:44:07 on dist-test-slave-lnj3"
server_key: "cf0e12351435cc3033292b13451d8c50"
server_key_iv: "fb612a3304919705cae001fdac940d70"
server_key_version: "encryptionkey@0"
I20260618 22:44:07.538293 28376 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.002s
I20260618 22:44:07.543243 28391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:44:07.545004 28376 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260618 22:44:07.545185 28376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "f61e8a5bbddb4cb78ec5970b500b75b1"
format_stamp: "Formatted at 2026-06-18 22:44:07 on dist-test-slave-lnj3"
server_key: "cf0e12351435cc3033292b13451d8c50"
server_key_iv: "fb612a3304919705cae001fdac940d70"
server_key_version: "encryptionkey@0"
I20260618 22:44:07.545440 28376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:44:07.566062 28376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:44:07.571439 28376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:44:07.571724 28376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:44:07.590149 28376 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:40785
I20260618 22:44:07.590162 28442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:40785 every 8 connection(s)
I20260618 22:44:07.591521 28376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:44:07.595096 28443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:44:07.598531 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 28376
I20260618 22:44:07.598758 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:44:07.599555 5975 external_mini_cluster.cc:1428] Setting key e524381f3e1fe61a190301396f37a67a
I20260618 22:44:07.602989 28443 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:44:07 dist-test-slave-lnj3 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822647, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.213.253@KRBTEST.COM; defaulting to no policy
I20260618 22:44:07.627988 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:07.607789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.5.213.254:40785)
0618 22:44:07.608382 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:07.608391 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:07.608436 (+ 45us) server_negotiation.cc:407] Connection header received
0618 22:44:07.610287 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:07.610296 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:07.610692 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:07.610901 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:07.612546 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:07.613797 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:07.615741 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:07.616359 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:07.616497 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:07.620425 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:07.620434 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:07.620447 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:07.620480 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:07.623380 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:07.624309 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:07.624316 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:07.624318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:07.624479 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:07.624965 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:07.624969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:07.624972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:07.625505 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:07.625674 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:07.626409 (+ 735us) server_negotiation.cc:299] Negotiation successful
0618 22:44:07.626752 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":436,"thread_start_us":288,"threads_started":1}
I20260618 22:44:07.628005 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:07.606522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:40785 (local address 127.0.0.1:53974)
0618 22:44:07.607380 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:07.607435 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0618 22:44:07.609858 (+ 2423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:44:07.610974 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:44:07.610985 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:44:07.611832 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:44:07.612294 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:07.612319 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:07.613963 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:07.613968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:07.615585 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:07.615600 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:07.616601 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:07.616608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:07.616804 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:07.617487 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:44:07.617516 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:44:07.620182 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:44:07.623518 (+ 3336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:07.623532 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:07.623547 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:07.624115 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:07.624588 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:07.624593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:07.624597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:07.624818 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:07.625665 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:44:07.625673 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:44:07.625926 (+ 253us) client_negotiation.cc:769] Sending connection context
0618 22:44:07.626231 (+ 305us) client_negotiation.cc:240] Negotiation successful
0618 22:44:07.626916 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":200,"thread_start_us":307,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.253@KRBTEST.COM" created.
I20260618 22:44:07.631037 28443 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:44:07.632157 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:07.631771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:39699 (local address 127.0.0.1:49496)
0618 22:44:07.631935 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:07.632043 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:39699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260618 22:44:07.632438 28393 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.213.253:39699: connect: Connection refused (error 111)
I20260618 22:44:07.633947 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:07.633563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:39699 (local address 127.0.0.1:49504)
0618 22:44:07.633745 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:07.633819 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:39699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260618 22:44:07.634135 28443 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.253:39699: Network error: Client connection negotiation failed: client connection to 127.5.213.253:39699: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.253.keytab.
Entry for principal kudu/127.5.213.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.253.keytab.
I20260618 22:44:07.657568 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.253:39699
--webserver_interface=127.5.213.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.253.keytab
--principal=kudu/127.5.213.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:40785,127.5.213.253:39699,127.5.213.252:41315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:44:07.683455 28443 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } attempt: 1
I20260618 22:44:07.684535 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:07.684082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:39699 (local address 127.0.0.1:49520)
0618 22:44:07.684292 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:07.684405 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:39699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260618 22:44:07.685529 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:07.685187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:39699 (local address 127.0.0.1:49522)
0618 22:44:07.685337 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:07.685403 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:39699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260618 22:44:07.685696 28443 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.253:39699: Network error: Client connection negotiation failed: client connection to 127.5.213.253:39699: connect: Connection refused (error 111)
I20260618 22:44:07.753930 28443 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } attempt: 2
I20260618 22:44:07.755041 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:07.754546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:39699 (local address 127.0.0.1:49526)
0618 22:44:07.754809 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:07.754906 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:39699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260618 22:44:07.755932 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:07.755602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:39699 (local address 127.0.0.1:49540)
0618 22:44:07.755735 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:07.755813 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:39699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260618 22:44:07.756094 28443 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.253:39699: Network error: Client connection negotiation failed: client connection to 127.5.213.253:39699: connect: Connection refused (error 111)
I20260618 22:44:07.847332 28443 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } attempt: 3
I20260618 22:44:07.848608 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:07.848127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:39699 (local address 127.0.0.1:49542)
0618 22:44:07.848359 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:07.848460 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:39699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260618 22:44:07.849614 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:07.849242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:39699 (local address 127.0.0.1:49550)
0618 22:44:07.849385 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:07.849458 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:39699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260618 22:44:07.849798 28443 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.253:39699: Network error: Client connection negotiation failed: client connection to 127.5.213.253:39699: connect: Connection refused (error 111)
W20260618 22:44:07.989176 28450 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 22:44:07.989555 28450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:44:07.989667 28450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:44:07.989805 28450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260618 22:44:07.993037 28443 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } attempt: 4
I20260618 22:44:07.994202 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:07.993630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:39699 (local address 127.0.0.1:49564)
0618 22:44:07.993961 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:07.994051 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:39699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260618 22:44:07.995406 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:07.995029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:39699 (local address 127.0.0.1:49576)
0618 22:44:07.995191 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:07.995277 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:39699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260618 22:44:07.995564 28443 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.253:39699: Network error: Client connection negotiation failed: client connection to 127.5.213.253:39699: connect: Connection refused (error 111)
W20260618 22:44:07.997575 28450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:44:07.997680 28450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:44:07.997778 28450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:44:07.997850 28450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:44:07.997939 28450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:44:08.008329 28450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:40785,127.5.213.253:39699,127.5.213.252:41315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.253.keytab
--principal=kudu/127.5.213.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.253:39699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.213.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:44:08.009694 28450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:44:08.011139 28450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:44:08.022850 28458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:44:08.022886 28456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:44:08.023102 28455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:44:08.024065 28450 server_base.cc:1034] running on GCE node
I20260618 22:44:08.024832 28450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:44:08.026186 28450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:44:08.027395 28450 hybrid_clock.cc:648] HybridClock initialized: now 1781822648027328 us; error 40 us; skew 500 ppm
I20260618 22:44:08.027705 28450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:44:08 dist-test-slave-lnj3 krb5kdc[28363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822648, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:44:08.030725 28450 init.cc:377] Logged in from keytab as kudu/127.5.213.253@KRBTEST.COM (short username kudu)
I20260618 22:44:08.032100 28450 webserver.cc:460] Webserver started at http://127.5.213.253:40979/ using document root <none> and password file <none>
I20260618 22:44:08.032773 28450 fs_manager.cc:359] Metadata directory not provided
I20260618 22:44:08.032857 28450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:44:08.033139 28450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:44:08.036036 28450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-1/data/instance:
uuid: "f88d77755c0a4e6ba596adc0416da631"
format_stamp: "Formatted at 2026-06-18 22:44:08 on dist-test-slave-lnj3"
server_key: "aefeeea818f9077db21651e844bfab26"
server_key_iv: "632b1339c9e38954ccbca544b8200c59"
server_key_version: "encryptionkey@0"
I20260618 22:44:08.036774 28450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-1/wal/instance:
uuid: "f88d77755c0a4e6ba596adc0416da631"
format_stamp: "Formatted at 2026-06-18 22:44:08 on dist-test-slave-lnj3"
server_key: "aefeeea818f9077db21651e844bfab26"
server_key_iv: "632b1339c9e38954ccbca544b8200c59"
server_key_version: "encryptionkey@0"
I20260618 22:44:08.041679 28450 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260618 22:44:08.045022 28465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:44:08.046407 28450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:44:08.046561 28450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-1/wal
uuid: "f88d77755c0a4e6ba596adc0416da631"
format_stamp: "Formatted at 2026-06-18 22:44:08 on dist-test-slave-lnj3"
server_key: "aefeeea818f9077db21651e844bfab26"
server_key_iv: "632b1339c9e38954ccbca544b8200c59"
server_key_version: "encryptionkey@0"
I20260618 22:44:08.046803 28450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:44:08.064810 28450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:44:08.069059 28450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:44:08.069324 28450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:44:08.086676 28450 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.253:39699
I20260618 22:44:08.086715 28516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.253:39699 every 8 connection(s)
I20260618 22:44:08.088004 28450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-1/data/info.pb
I20260618 22:44:08.090538 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 28450
I20260618 22:44:08.090723 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-1/wal/instance
I20260618 22:44:08.091126 5975 external_mini_cluster.cc:1428] Setting key 84d4c48232d32d57983c7bc26e95810c
I20260618 22:44:08.091708 28517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:44:08.098342 28517 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.213.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.252@KRBTEST.COM" created.
Jun 18 22:44:08 dist-test-slave-lnj3 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822648, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.253@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:44:08.122416 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.101013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53980 (local address 127.5.213.254:40785)
0618 22:44:08.101219 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:08.101225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:08.102519 (+ 1294us) server_negotiation.cc:407] Connection header received
0618 22:44:08.104424 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:08.104430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:08.104510 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:08.104613 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:08.106074 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.107155 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.108963 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.109603 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.109739 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.115402 (+ 5663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:08.115408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:08.115411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:08.115447 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:08.118196 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.119043 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.119048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.119050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.119180 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.119743 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.119750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.119753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.120155 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:08.120327 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:08.121846 (+ 1519us) server_negotiation.cc:299] Negotiation successful
0618 22:44:08.122081 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260618 22:44:08.123129 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.101522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:40785 (local address 127.0.0.1:53980)
0618 22:44:08.102344 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.102387 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 22:44:08.104152 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:44:08.104813 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:44:08.104822 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:44:08.105355 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:44:08.105915 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:08.105932 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.107339 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.107344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:08.108801 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:08.108813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.109837 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.109843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:08.110053 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.110802 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:44:08.110826 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:44:08.115161 (+ 4335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:44:08.118372 (+ 3211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:08.118381 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:08.118396 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:08.118917 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:08.119297 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:08.119301 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:08.119303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:08.119535 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:08.120342 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:44:08.120352 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:44:08.120817 (+ 465us) client_negotiation.cc:769] Sending connection context
0618 22:44:08.121101 (+ 284us) client_negotiation.cc:240] Negotiation successful
0618 22:44:08.121660 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":336,"threads_started":1}
I20260618 22:44:08.124845 28517 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:44:08 dist-test-slave-lnj3 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822648, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.253@KRBTEST.COM for kudu/127.5.213.253@KRBTEST.COM
I20260618 22:44:08.140806 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.125651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:39699 (local address 127.0.0.1:49592)
0618 22:44:08.125894 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.125914 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:44:08.126029 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:44:08.127164 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:44:08.127167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:44:08.127537 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:44:08.127687 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:08.127695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.129498 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.129502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:08.130416 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:08.130428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.131491 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.131496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:08.131635 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.132195 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:44:08.132213 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:44:08.134435 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:44:08.138366 (+ 3931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:08.138370 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:08.138373 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:08.138774 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:08.139164 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:08.139167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:08.139169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:08.139299 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:08.140046 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:44:08.140050 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:44:08.140175 (+ 125us) client_negotiation.cc:769] Sending connection context
0618 22:44:08.140287 (+ 112us) client_negotiation.cc:240] Negotiation successful
0618 22:44:08.140523 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260618 22:44:08.141153 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.125864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49592 (local address 127.5.213.253:39699)
0618 22:44:08.126601 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:08.126609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:08.126712 (+ 103us) server_negotiation.cc:407] Connection header received
0618 22:44:08.126766 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:08.126773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:08.127012 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:08.127147 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:08.127859 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.129361 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.130549 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.131353 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.131468 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.134695 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:08.134712 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:08.134733 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:08.134781 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:08.138240 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.138881 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.138888 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.138890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.139038 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.139393 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.139400 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.139402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.139901 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:08.140085 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:08.140452 (+ 367us) server_negotiation.cc:299] Negotiation successful
0618 22:44:08.140639 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":427,"threads_started":1}
I20260618 22:44:08.143635 28517 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.252.keytab.
Entry for principal kudu/127.5.213.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.252.keytab.
I20260618 22:44:08.144783 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.144277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52730)
0618 22:44:08.144438 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.144561 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260618 22:44:08.145004 28467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
I20260618 22:44:08.146054 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.252:41315
--webserver_interface=127.5.213.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.252.keytab
--principal=kudu/127.5.213.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:40785,127.5.213.253:39699,127.5.213.252:41315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:44:08.146608 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.146282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52732)
0618 22:44:08.146420 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.146503 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260618 22:44:08.146936 28517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:41315: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
I20260618 22:44:08.196236 28517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } attempt: 1
I20260618 22:44:08.197381 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.196925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52740)
0618 22:44:08.197120 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.197227 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260618 22:44:08.198390 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.198039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52746)
0618 22:44:08.198178 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.198285 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260618 22:44:08.198603 28517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:41315: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
I20260618 22:44:08.266855 28517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } attempt: 2
I20260618 22:44:08.267902 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.267474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52760)
0618 22:44:08.267660 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.267753 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260618 22:44:08.268904 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.268598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52776)
0618 22:44:08.268723 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.268796 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260618 22:44:08.269097 28517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:41315: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
I20260618 22:44:08.294842 28443 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } attempt: 5
Jun 18 22:44:08 dist-test-slave-lnj3 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822647, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for kudu/127.5.213.253@KRBTEST.COM
I20260618 22:44:08.310020 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.295739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49596 (local address 127.5.213.253:39699)
0618 22:44:08.295935 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:08.295939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:08.295977 (+ 38us) server_negotiation.cc:407] Connection header received
0618 22:44:08.296199 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:08.296204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:08.296296 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:08.296388 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:08.297489 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.298291 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.300088 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.300544 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.300700 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.304636 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:08.304641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:08.304645 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:08.304682 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:08.307364 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.308141 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.308146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.308148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.308273 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.308730 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.308734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.308736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.309040 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:08.309165 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:08.309580 (+ 415us) server_negotiation.cc:299] Negotiation successful
0618 22:44:08.309801 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260618 22:44:08.310139 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.295663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:39699 (local address 127.0.0.1:49596)
0618 22:44:08.295885 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.295907 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0618 22:44:08.296038 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:44:08.296467 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:44:08.296473 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:44:08.297006 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:44:08.297301 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:08.297317 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.298455 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.298461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:08.299915 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:08.299932 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.300744 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.300751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:08.300981 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.301636 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:44:08.301661 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:44:08.304388 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:44:08.307504 (+ 3116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:08.307509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:08.307513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:08.308013 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:08.308404 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:08.308408 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:08.308412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:08.308579 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:08.309153 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:44:08.309158 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:44:08.309332 (+ 174us) client_negotiation.cc:769] Sending connection context
0618 22:44:08.309486 (+ 154us) client_negotiation.cc:240] Negotiation successful
0618 22:44:08.309810 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260618 22:44:08.311172 28443 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:44:08.312182 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.311767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52788)
0618 22:44:08.311943 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.312022 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260618 22:44:08.313099 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.312806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52802)
0618 22:44:08.312924 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.313001 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260618 22:44:08.313284 28443 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:41315: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
I20260618 22:44:08.360311 28517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } attempt: 3
I20260618 22:44:08.361545 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.361063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52816)
0618 22:44:08.361287 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.361391 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260618 22:44:08.362581 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.362166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52826)
0618 22:44:08.362384 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.362465 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260618 22:44:08.362799 28517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:41315: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
I20260618 22:44:08.364494 28443 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } attempt: 1
I20260618 22:44:08.365486 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.365096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52832)
0618 22:44:08.365267 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.365347 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260618 22:44:08.366343 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.366054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52836)
0618 22:44:08.366172 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.366245 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260618 22:44:08.366575 28443 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:41315: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
I20260618 22:44:08.434803 28443 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } attempt: 2
I20260618 22:44:08.436039 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.435523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52846)
0618 22:44:08.435791 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.435880 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260618 22:44:08.437181 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.436737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52852)
0618 22:44:08.436967 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.437045 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260618 22:44:08.437376 28443 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:41315: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
W20260618 22:44:08.470646 28524 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 22:44:08.470949 28524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:44:08.471016 28524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:44:08.471093 28524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:44:08.478080 28524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:44:08.478168 28524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:44:08.478228 28524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:44:08.478277 28524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:44:08.478334 28524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:44:08.487902 28524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:40785,127.5.213.253:39699,127.5.213.252:41315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.252.keytab
--principal=kudu/127.5.213.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.252:41315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.213.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:44:08.489149 28524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:44:08.490419 28524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:44:08.501493 28532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:44:08.501590 28530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:44:08.501521 28529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:44:08.502434 28524 server_base.cc:1034] running on GCE node
I20260618 22:44:08.503216 28524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:44:08.504544 28524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:44:08.505761 28524 hybrid_clock.cc:648] HybridClock initialized: now 1781822648505689 us; error 43 us; skew 500 ppm
I20260618 22:44:08.506059 28524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260618 22:44:08.505990 28517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } attempt: 4
I20260618 22:44:08.506942 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.506495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52860)
0618 22:44:08.506723 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.506801 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260618 22:44:08.507884 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.507575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52868)
0618 22:44:08.507707 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.507780 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260618 22:44:08.508062 28517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:41315: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
Jun 18 22:44:08 dist-test-slave-lnj3 krb5kdc[28363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822648, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:44:08.509433 28524 init.cc:377] Logged in from keytab as kudu/127.5.213.252@KRBTEST.COM (short username kudu)
I20260618 22:44:08.510846 28524 webserver.cc:460] Webserver started at http://127.5.213.252:43045/ using document root <none> and password file <none>
I20260618 22:44:08.511495 28524 fs_manager.cc:359] Metadata directory not provided
I20260618 22:44:08.511583 28524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:44:08.511880 28524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:44:08.514766 28524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-2/data/instance:
uuid: "0e7fbff6d25f40928bf940da426b8201"
format_stamp: "Formatted at 2026-06-18 22:44:08 on dist-test-slave-lnj3"
server_key: "a89f76d75c3902ec408a5579c1158d7e"
server_key_iv: "20aa5ee8b4bb17e83af1fde5dd0a4c2b"
server_key_version: "encryptionkey@0"
I20260618 22:44:08.515532 28524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-2/wal/instance:
uuid: "0e7fbff6d25f40928bf940da426b8201"
format_stamp: "Formatted at 2026-06-18 22:44:08 on dist-test-slave-lnj3"
server_key: "a89f76d75c3902ec408a5579c1158d7e"
server_key_iv: "20aa5ee8b4bb17e83af1fde5dd0a4c2b"
server_key_version: "encryptionkey@0"
I20260618 22:44:08.520623 28524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:44:08.523921 28539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:44:08.525195 28524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:44:08.525341 28524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-2/wal
uuid: "0e7fbff6d25f40928bf940da426b8201"
format_stamp: "Formatted at 2026-06-18 22:44:08 on dist-test-slave-lnj3"
server_key: "a89f76d75c3902ec408a5579c1158d7e"
server_key_iv: "20aa5ee8b4bb17e83af1fde5dd0a4c2b"
server_key_version: "encryptionkey@0"
I20260618 22:44:08.525528 28524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:44:08.543613 28443 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } attempt: 3
I20260618 22:44:08.544746 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.544276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52882)
0618 22:44:08.544480 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.544571 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260618 22:44:08.545637 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.545316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52884)
0618 22:44:08.545472 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.545543 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260618 22:44:08.545867 28443 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:41315: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41315: connect: Connection refused (error 111)
I20260618 22:44:08.551403 28524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:44:08.555173 28524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:44:08.555405 28524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:44:08.571484 28524 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.252:41315
I20260618 22:44:08.571465 28590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.252:41315 every 8 connection(s)
I20260618 22:44:08.572794 28524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-2/data/info.pb
I20260618 22:44:08.576433 28591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:44:08.579988 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 28524
I20260618 22:44:08.580163 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781822307836422-5975-0/minicluster-data/master-2/wal/instance
I20260618 22:44:08.580669 5975 external_mini_cluster.cc:1428] Setting key 82b55cfd761328c66aa07f53eb3fa754
I20260618 22:44:08.584162 28591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:44:08 dist-test-slave-lnj3 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822648, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.252@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
Jun 18 22:44:08 dist-test-slave-lnj3 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.252@KRBTEST.COM
I20260618 22:44:08.604820 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.586525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.5.213.254:40785)
0618 22:44:08.586744 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:08.586751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:08.587276 (+ 525us) server_negotiation.cc:407] Connection header received
0618 22:44:08.589053 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:08.589060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:08.589148 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:08.589271 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:08.591231 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.592125 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.594011 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.594726 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.594843 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.598888 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:08.598892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:08.598896 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:08.598929 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:08.601794 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.602522 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.602528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.602532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.602699 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.603100 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.603105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.603109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.603381 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:08.603506 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:08.604264 (+ 758us) server_negotiation.cc:299] Negotiation successful
0618 22:44:08.604489 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260618 22:44:08.605942 28595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.586426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:40785 (local address 127.0.0.1:53982)
0618 22:44:08.587128 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.587169 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 22:44:08.588818 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:44:08.589521 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:44:08.589533 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:44:08.590234 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:44:08.591008 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:08.591031 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.592292 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.592296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:08.593866 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:08.593881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.594963 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.594971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:08.595297 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.596095 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:44:08.596126 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:44:08.598705 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:44:08.601906 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:08.601921 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:08.601930 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:08.602398 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:08.602804 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:08.602807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:08.602810 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:08.602985 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:08.603498 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:44:08.603505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:44:08.603884 (+ 379us) client_negotiation.cc:769] Sending connection context
0618 22:44:08.604108 (+ 224us) client_negotiation.cc:240] Negotiation successful
0618 22:44:08.604598 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":58,"thread_start_us":349,"threads_started":1}
I20260618 22:44:08.607316 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.582683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.5.213.252:41315)
0618 22:44:08.583502 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:08.583517 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:08.590998 (+ 7481us) server_negotiation.cc:407] Connection header received
0618 22:44:08.591150 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:08.591156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:08.591385 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:08.591546 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:08.593164 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.594479 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.596119 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.597016 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.597131 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.601663 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:08.601675 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:08.601691 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:08.601745 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:08.604485 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.605145 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.605154 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.605158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.605311 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.605713 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.605719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.605722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.606111 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:08.606431 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:08.606704 (+ 273us) server_negotiation.cc:299] Negotiation successful
0618 22:44:08.606933 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":565,"thread_start_us":325,"threads_started":1}
I20260618 22:44:08.607858 28591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:44:08 dist-test-slave-lnj3 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822648, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.252@KRBTEST.COM for kudu/127.5.213.253@KRBTEST.COM
Jun 18 22:44:08 dist-test-slave-lnj3 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:44:08.621134 28595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.608613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:39699 (local address 127.0.0.1:49600)
0618 22:44:08.608775 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.608793 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:44:08.608886 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:44:08.609387 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:44:08.609392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:44:08.609792 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:44:08.609959 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:08.609968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.610998 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.611002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:08.611878 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:08.611889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.612548 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.612553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:08.612724 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.613263 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:44:08.613279 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:44:08.615572 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:44:08.618793 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:08.618797 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:08.618799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:08.619219 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:08.619684 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:08.619688 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:08.619690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:08.619820 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:08.620403 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:44:08.620406 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:44:08.620538 (+ 132us) client_negotiation.cc:769] Sending connection context
0618 22:44:08.620695 (+ 157us) client_negotiation.cc:240] Negotiation successful
0618 22:44:08.620896 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260618 22:44:08.621557 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.608681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.5.213.253:39699)
0618 22:44:08.608856 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:08.608860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:08.608876 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:44:08.609027 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:08.609033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:08.609121 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:08.609189 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:08.610108 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.610871 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.612007 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.612420 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.612525 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.615800 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:08.615805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:08.615808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:08.615842 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:08.618565 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.619337 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.619343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.619346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.619513 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.619922 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.619927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.619930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.620274 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:08.620430 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:08.620793 (+ 363us) server_negotiation.cc:299] Negotiation successful
0618 22:44:08.621000 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260618 22:44:08.622517 28591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:44:08.624337 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.610167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.5.213.254:40785)
0618 22:44:08.610314 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:08.610319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:08.610335 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:44:08.610486 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:08.610492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:08.610554 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:08.610659 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:08.611777 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.612500 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.613738 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.614142 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.614247 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.618447 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:08.618453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:08.618458 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:08.618498 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:08.621592 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.622675 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.622679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.622681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.622840 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.623183 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.623188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.623191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.623544 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:08.623787 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:08.623981 (+ 194us) server_negotiation.cc:299] Negotiation successful
0618 22:44:08.624135 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Jun 18 22:44:08 dist-test-slave-lnj3 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822648, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.252@KRBTEST.COM for kudu/127.5.213.252@KRBTEST.COM
Jun 18 22:44:08 dist-test-slave-lnj3 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.253@KRBTEST.COM
I20260618 22:44:08.637102 28595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.623176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52904)
0618 22:44:08.623421 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.623438 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:44:08.623543 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:44:08.624658 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:44:08.624662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:44:08.625024 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:44:08.625179 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:08.625191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.626247 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.626252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:08.627592 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:08.627612 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.628487 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.628493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:08.628664 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.629144 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:44:08.629166 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:44:08.631586 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:44:08.634753 (+ 3167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:08.634758 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:08.634762 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:08.635251 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:08.635555 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:08.635560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:08.635563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:08.635723 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:08.636279 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:44:08.636283 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:44:08.636429 (+ 146us) client_negotiation.cc:769] Sending connection context
0618 22:44:08.636565 (+ 136us) client_negotiation.cc:240] Negotiation successful
0618 22:44:08.636831 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260618 22:44:08.637326 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.623339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.5.213.252:41315)
0618 22:44:08.623485 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:08.623488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:08.623502 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:44:08.624351 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:08.624357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:08.624424 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:08.624506 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:08.625311 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.626116 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.627738 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.628346 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.628497 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.631756 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:08.631760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:08.631763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:08.631790 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:08.634580 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.635351 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.635354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.635357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.635457 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.635823 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.635828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.635830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.636139 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:08.636277 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:08.636691 (+ 414us) server_negotiation.cc:299] Negotiation successful
0618 22:44:08.636977 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":30}
I20260618 22:44:08.641667 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.625757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49608 (local address 127.5.213.253:39699)
0618 22:44:08.625903 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:08.625908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:08.625924 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:44:08.625995 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:08.626000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:08.626059 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:08.626150 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:08.627735 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.628762 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.630440 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.631082 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.631186 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.635919 (+ 4733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:08.635925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:08.635929 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:08.635963 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:08.639305 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.639915 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.639920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.639923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.640075 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.640368 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.640372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.640375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.640697 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:08.640942 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:08.641129 (+ 187us) server_negotiation.cc:299] Negotiation successful
0618 22:44:08.641361 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 22:44:08.643047 28591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201: Bootstrap starting.
I20260618 22:44:08.646400 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:44:08.646471 28591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201: Neither blocks nor log segments found. Creating new log.
I20260618 22:44:08.647511 28591 log.cc:824] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201: Log is configured to *not* fsync() on all Append() calls
I20260618 22:44:08.650580 28591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201: No bootstrap required, opened a new log
I20260618 22:44:08.649137 28418 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53992
I20260618 22:44:08.661325 28591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } } peers { permanent_uuid: "f88d77755c0a4e6ba596adc0416da631" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } } peers { permanent_uuid: "0e7fbff6d25f40928bf940da426b8201" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } }
I20260618 22:44:08.661769 28591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e7fbff6d25f40928bf940da426b8201, State: Initialized, Role: FOLLOWER
I20260618 22:44:08.662262 28591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } } peers { permanent_uuid: "f88d77755c0a4e6ba596adc0416da631" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } } peers { permanent_uuid: "0e7fbff6d25f40928bf940da426b8201" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } }
I20260618 22:44:08.663264 28597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } } peers { permanent_uuid: "f88d77755c0a4e6ba596adc0416da631" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } } peers { permanent_uuid: "0e7fbff6d25f40928bf940da426b8201" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } } }
I20260618 22:44:08.663559 28597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:44:08.664522 28591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201 [sys.catalog]: configured and running, proceeding with master startup.
W20260618 22:44:08.673276 28608 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:44:08.673400 28608 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:44:08.671267 28524 master_runner.cc:186] Error getting master registration for 127.5.213.254:40785: OK, instance_id { permanent_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" instance_seqno: 1781822647585115 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 22:44:08.681921 28524 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 22:44:08.723052 28443 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } attempt: 4
Jun 18 22:44:08 dist-test-slave-lnj3 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822647, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for kudu/127.5.213.252@KRBTEST.COM
I20260618 22:44:08.736491 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.723812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.5.213.252:41315)
0618 22:44:08.724010 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:08.724015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:08.724036 (+ 21us) server_negotiation.cc:407] Connection header received
0618 22:44:08.724146 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:08.724151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:08.724240 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:08.724321 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:08.725106 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.725852 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.727196 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.727631 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.727738 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.731384 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:08.731389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:08.731393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:08.731424 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:08.733970 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.734551 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.734555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.734557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.734710 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.735057 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.735062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.735066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.735423 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:08.735576 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:08.735918 (+ 342us) server_negotiation.cc:299] Negotiation successful
0618 22:44:08.736115 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260618 22:44:08.736488 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.723712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52906)
0618 22:44:08.723889 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.723910 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:44:08.724033 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:44:08.724379 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:44:08.724384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:44:08.724793 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:44:08.724956 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:08.724966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.725992 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.725998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:08.727050 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:08.727065 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.727777 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.727782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:08.727980 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.728544 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:44:08.728576 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:44:08.731201 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:44:08.734087 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:08.734093 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:08.734097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:08.734446 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:08.734808 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:08.734814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:08.734818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:08.734958 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:08.735533 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:44:08.735537 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:44:08.735690 (+ 153us) client_negotiation.cc:769] Sending connection context
0618 22:44:08.735813 (+ 123us) client_negotiation.cc:240] Negotiation successful
0618 22:44:08.736079 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260618 22:44:08.741029 28443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1: Bootstrap starting.
I20260618 22:44:08.744002 28443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1: Neither blocks nor log segments found. Creating new log.
I20260618 22:44:08.744989 28443 log.cc:824] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1: Log is configured to *not* fsync() on all Append() calls
I20260618 22:44:08.747599 28443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1: No bootstrap required, opened a new log
I20260618 22:44:08.753945 28443 raft_consensus.cc:348] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } } peers { permanent_uuid: "f88d77755c0a4e6ba596adc0416da631" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } } peers { permanent_uuid: "0e7fbff6d25f40928bf940da426b8201" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } }
I20260618 22:44:08.754315 28443 raft_consensus.cc:729] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61e8a5bbddb4cb78ec5970b500b75b1, State: Initialized, Role: FOLLOWER
I20260618 22:44:08.754864 28443 consensus_queue.cc:260] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } } peers { permanent_uuid: "f88d77755c0a4e6ba596adc0416da631" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } } peers { permanent_uuid: "0e7fbff6d25f40928bf940da426b8201" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } }
I20260618 22:44:08.755955 28611 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } } peers { permanent_uuid: "f88d77755c0a4e6ba596adc0416da631" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } } peers { permanent_uuid: "0e7fbff6d25f40928bf940da426b8201" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } } }
I20260618 22:44:08.756579 28611 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:44:08.757155 28443 sys_catalog.cc:564] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:44:08.757259 28418 raft_consensus.cc:482] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260618 22:44:08.757424 28418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:44:08.760874 28418 raft_consensus.cc:504] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } } peers { permanent_uuid: "f88d77755c0a4e6ba596adc0416da631" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } } peers { permanent_uuid: "0e7fbff6d25f40928bf940da426b8201" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } }
I20260618 22:44:08.762390 28418 leader_election.cc:288] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [CANDIDATE]: Term 1 election: Requested vote from peers f88d77755c0a4e6ba596adc0416da631 (127.5.213.253:39699), 0e7fbff6d25f40928bf940da426b8201 (127.5.213.252:41315)
I20260618 22:44:08.764022 28566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0e7fbff6d25f40928bf940da426b8201"
I20260618 22:44:08.764540 28566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:44:08.768198 28566 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f61e8a5bbddb4cb78ec5970b500b75b1 in term 1.
I20260618 22:44:08.768898 28393 leader_election.cc:302] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e7fbff6d25f40928bf940da426b8201, f61e8a5bbddb4cb78ec5970b500b75b1; no voters:
I20260618 22:44:08.769400 28611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:44:08.781181 28611 raft_consensus.cc:686] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [term 1 LEADER]: Becoming Leader. State: Replica: f61e8a5bbddb4cb78ec5970b500b75b1, State: Running, Role: LEADER
I20260618 22:44:08.782362 28611 consensus_queue.cc:237] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } } peers { permanent_uuid: "f88d77755c0a4e6ba596adc0416da631" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } } peers { permanent_uuid: "0e7fbff6d25f40928bf940da426b8201" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } }
I20260618 22:44:08.793816 28619 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f61e8a5bbddb4cb78ec5970b500b75b1. Latest consensus state: current_term: 1 leader_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } } peers { permanent_uuid: "f88d77755c0a4e6ba596adc0416da631" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } } peers { permanent_uuid: "0e7fbff6d25f40928bf940da426b8201" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } } }
I20260618 22:44:08.794595 28619 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [sys.catalog]: This master's current role is: LEADER
I20260618 22:44:08.807199 28517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } attempt: 5
W20260618 22:44:08.807416 28545 tablet.cc:2369] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 18 22:44:08 dist-test-slave-lnj3 krb5kdc[28363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822648, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.253@KRBTEST.COM for kudu/127.5.213.252@KRBTEST.COM
W20260618 22:44:08.824847 28397 tablet.cc:2369] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:44:08.826372 28625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:44:08.829221 28633 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260618 22:44:08.833907 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.807801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41315 (local address 127.0.0.1:52910)
0618 22:44:08.808152 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:08.808172 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:44:08.808308 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:44:08.809166 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:44:08.809171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:44:08.809908 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:44:08.810137 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:08.810151 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.811823 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.811830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:08.813332 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:08.813349 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.814761 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.814768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:08.815186 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.816154 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:44:08.816180 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:44:08.820102 (+ 3922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:44:08.829833 (+ 9731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:08.829853 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:08.829857 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:08.830413 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:08.831120 (+ 707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:08.831125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:08.831129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:08.831395 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:08.832424 (+ 1029us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:44:08.832430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:44:08.832710 (+ 280us) client_negotiation.cc:769] Sending connection context
0618 22:44:08.832922 (+ 212us) client_negotiation.cc:240] Negotiation successful
0618 22:44:08.833378 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234}
I20260618 22:44:08.835839 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.808524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52910 (local address 127.5.213.252:41315)
0618 22:44:08.808691 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:08.808697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:08.808715 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:44:08.808782 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:08.808787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:08.808860 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:08.808951 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:08.810368 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.811614 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.813564 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.814356 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.814474 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.820423 (+ 5949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:08.820430 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:08.820435 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:08.820480 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:08.829180 (+ 8700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.830665 (+ 1485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.830672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.830676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.830944 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.831637 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.831645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.831648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.832239 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:08.834669 (+ 2430us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:08.834875 (+ 206us) server_negotiation.cc:299] Negotiation successful
0618 22:44:08.835268 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 22:44:08.841602 28517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f88d77755c0a4e6ba596adc0416da631: Bootstrap starting.
I20260618 22:44:08.833240 28376 master_runner.cc:186] Error getting master registration for 127.5.213.253:39699: OK, instance_id { permanent_uuid: "f88d77755c0a4e6ba596adc0416da631" instance_seqno: 1781822648081716 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 22:44:08.843477 28376 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 22:44:08.846873 28625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:44:08.847584 28517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f88d77755c0a4e6ba596adc0416da631: Neither blocks nor log segments found. Creating new log.
I20260618 22:44:08.849466 28517 log.cc:824] T 00000000000000000000000000000000 P f88d77755c0a4e6ba596adc0416da631: Log is configured to *not* fsync() on all Append() calls
I20260618 22:44:08.862743 28517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f88d77755c0a4e6ba596adc0416da631: No bootstrap required, opened a new log
I20260618 22:44:08.876982 28566 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201 [term 1 FOLLOWER]: Refusing update from remote peer f61e8a5bbddb4cb78ec5970b500b75b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:44:08.877849 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.803872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.5.213.254:40785)
0618 22:44:08.804054 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:08.804060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:08.820925 (+ 16865us) server_negotiation.cc:407] Connection header received
0618 22:44:08.823296 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:08.823303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:08.823400 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:08.823475 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:08.835019 (+ 11544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.836137 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.851161 (+ 15024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.851920 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.852046 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.866419 (+ 14373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:08.866427 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:08.866431 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:08.866492 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:08.870183 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.871791 (+ 1608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.871796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.871800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.871975 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.875689 (+ 3714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.875694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.875697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.876069 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:08.876347 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:08.877256 (+ 909us) server_negotiation.cc:299] Negotiation successful
0618 22:44:08.877501 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260618 22:44:08.878461 28619 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e7fbff6d25f40928bf940da426b8201" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:44:08.880048 28630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.809584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52916 (local address 127.5.213.252:41315)
0618 22:44:08.822179 (+ 12595us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:08.822185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:08.823437 (+ 1252us) server_negotiation.cc:407] Connection header received
0618 22:44:08.828297 (+ 4860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:08.828303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:08.828395 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:08.828498 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:08.837869 (+ 9371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.839071 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.852264 (+ 13193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.853019 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.853152 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.859314 (+ 6162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:08.859322 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:08.859326 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:08.859373 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:08.865053 (+ 5680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.870846 (+ 5793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.870851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.870855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.871027 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.873791 (+ 2764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.873796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.873800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.874202 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:08.874439 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:08.879401 (+ 4962us) server_negotiation.cc:299] Negotiation successful
0618 22:44:08.879641 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12429,"thread_start_us":12590,"threads_started":1}
I20260618 22:44:08.889045 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:08.795528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49622 (local address 127.5.213.253:39699)
0618 22:44:08.795735 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:08.795741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:08.809025 (+ 13284us) server_negotiation.cc:407] Connection header received
0618 22:44:08.832922 (+ 23897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:08.832927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:08.832997 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:08.833055 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:08.849772 (+ 16717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.850972 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.861304 (+ 10332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:08.861930 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:08.862047 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:08.881101 (+ 19054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:08.881108 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:08.881112 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:08.881155 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:08.884841 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.885926 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.885931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.885935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.886096 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:08.886509 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:08.886513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:08.886517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:08.886920 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:08.887151 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:08.888421 (+ 1270us) server_negotiation.cc:299] Negotiation successful
0618 22:44:08.888660 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260618 22:44:08.903177 28517 raft_consensus.cc:348] T 00000000000000000000000000000000 P f88d77755c0a4e6ba596adc0416da631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } } peers { permanent_uuid: "f88d77755c0a4e6ba596adc0416da631" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } } peers { permanent_uuid: "0e7fbff6d25f40928bf940da426b8201" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } }
I20260618 22:44:08.905934 28597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f61e8a5bbddb4cb78ec5970b500b75b1. Latest consensus state: current_term: 1 leader_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } } peers { permanent_uuid: "f88d77755c0a4e6ba596adc0416da631" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } } peers { permanent_uuid: "0e7fbff6d25f40928bf940da426b8201" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } } }
I20260618 22:44:08.906229 28597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:44:08.906719 28517 raft_consensus.cc:729] T 00000000000000000000000000000000 P f88d77755c0a4e6ba596adc0416da631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f88d77755c0a4e6ba596adc0416da631, State: Initialized, Role: FOLLOWER
I20260618 22:44:08.908367 28636 mvcc.cc:205] Tried to move back new op lower bound from 7298345569779527680 to 7298345569423278080. Current Snapshot: MvccSnapshot[applied={T|T < 7298345569779527680}]
I20260618 22:44:08.911273 28597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } } peers { permanent_uuid: "f88d77755c0a4e6ba596adc0416da631" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } } peers { permanent_uuid: "0e7fbff6d25f40928bf940da426b8201" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } } }
I20260618 22:44:08.911500 28597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:44:08.914875 28637 mvcc.cc:205] Tried to move back new op lower bound from 7298345569779527680 to 7298345569423278080. Current Snapshot: MvccSnapshot[applied={T|T < 7298345569779527680}]
I20260618 22:44:08.917086 28517 consensus_queue.cc:260] T 00000000000000000000000000000000 P f88d77755c0a4e6ba596adc0416da631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } } peers { permanent_uuid: "f88d77755c0a4e6ba596adc0416da631" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } } peers { permanent_uuid: "0e7fbff6d25f40928bf940da426b8201" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } }
I20260618 22:44:08.918367 28641 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88d77755c0a4e6ba596adc0416da631 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } } peers { permanent_uuid: "f88d77755c0a4e6ba596adc0416da631" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } } peers { permanent_uuid: "0e7fbff6d25f40928bf940da426b8201" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } } }
I20260618 22:44:08.919152 28641 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88d77755c0a4e6ba596adc0416da631 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:44:08.920178 28517 sys_catalog.cc:564] T 00000000000000000000000000000000 P f88d77755c0a4e6ba596adc0416da631 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:44:08.920218 28492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f88d77755c0a4e6ba596adc0416da631"
I20260618 22:44:08.921036 28491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f88d77755c0a4e6ba596adc0416da631 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:44:08.925177 28491 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f88d77755c0a4e6ba596adc0416da631 [term 1 FOLLOWER]: Refusing update from remote peer f61e8a5bbddb4cb78ec5970b500b75b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:44:08.927105 28611 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } } peers { permanent_uuid: "f88d77755c0a4e6ba596adc0416da631" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } } peers { permanent_uuid: "0e7fbff6d25f40928bf940da426b8201" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } } }
I20260618 22:44:08.927446 28611 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [sys.catalog]: This master's current role is: LEADER
I20260618 22:44:08.928437 28625 catalog_manager.cc:1347] Generated new cluster ID: 9a34685b90a84f4691e8b873593824b5
I20260618 22:44:08.928530 28625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:44:08.929829 28492 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f88d77755c0a4e6ba596adc0416da631 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f61e8a5bbddb4cb78ec5970b500b75b1 in term 1.
I20260618 22:44:08.931435 28619 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f88d77755c0a4e6ba596adc0416da631" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:44:08.953747 28625 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:44:08.955474 28625 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:44:08.970041 28641 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88d77755c0a4e6ba596adc0416da631 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f61e8a5bbddb4cb78ec5970b500b75b1. Latest consensus state: current_term: 1 leader_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } } peers { permanent_uuid: "f88d77755c0a4e6ba596adc0416da631" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } } peers { permanent_uuid: "0e7fbff6d25f40928bf940da426b8201" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } } }
I20260618 22:44:08.970556 28641 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88d77755c0a4e6ba596adc0416da631 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:44:08.971552 28646 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88d77755c0a4e6ba596adc0416da631 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } } peers { permanent_uuid: "f88d77755c0a4e6ba596adc0416da631" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } } peers { permanent_uuid: "0e7fbff6d25f40928bf940da426b8201" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } } }
I20260618 22:44:08.971913 28611 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61e8a5bbddb4cb78ec5970b500b75b1" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 40785 } } peers { permanent_uuid: "f88d77755c0a4e6ba596adc0416da631" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 39699 } } peers { permanent_uuid: "0e7fbff6d25f40928bf940da426b8201" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41315 } } }
I20260618 22:44:08.972209 28611 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1 [sys.catalog]: This master's current role is: LEADER
I20260618 22:44:08.975641 28646 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88d77755c0a4e6ba596adc0416da631 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:44:08.979749 28625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f61e8a5bbddb4cb78ec5970b500b75b1: Generated new TSK 0
I20260618 22:44:08.980820 28625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:44:09.091188 28658 catalog_manager.cc:1259] Loaded cluster ID: 9a34685b90a84f4691e8b873593824b5
I20260618 22:44:09.098719 28658 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f88d77755c0a4e6ba596adc0416da631: loading cluster ID for follower catalog manager: success
I20260618 22:44:09.102936 28658 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f88d77755c0a4e6ba596adc0416da631: acquiring CA information for follower catalog manager: success
I20260618 22:44:09.105711 28658 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f88d77755c0a4e6ba596adc0416da631: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260618 22:44:09.126760 28522 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:09.110395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49636 (local address 127.5.213.253:39699)
0618 22:44:09.110598 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.110604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.122466 (+ 11862us) server_negotiation.cc:407] Connection header received
0618 22:44:09.125121 (+ 2655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.125128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.125230 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.125301 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:09.126496 (+ 1195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49636: BlockingRecv error: recv got EOF from 127.0.0.1:49636 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260618 22:44:09.132121 28630 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:09.123102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.5.213.252:41315)
0618 22:44:09.123322 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.123328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.128699 (+ 5371us) server_negotiation.cc:407] Connection header received
0618 22:44:09.130509 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.130516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.130601 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.130697 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:09.131852 (+ 1155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52924: BlockingRecv error: recv got EOF from 127.0.0.1:52924 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260618 22:44:09.133419 28446 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:09.118759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54002 (local address 127.5.213.254:40785)
0618 22:44:09.118939 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.118945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.127929 (+ 8984us) server_negotiation.cc:407] Connection header received
0618 22:44:09.129572 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.129578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.129693 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.129773 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:09.133170 (+ 3397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54002: BlockingRecv error: recv got EOF from 127.0.0.1:54002 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260618 22:44:09.182932 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:09.164751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49638 (local address 127.5.213.253:39699)
0618 22:44:09.164937 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.164943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.174985 (+ 10042us) server_negotiation.cc:407] Connection header received
0618 22:44:09.175374 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.175381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.175606 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.175766 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:44:09.176551 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.177709 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.179746 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.180424 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.180523 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:09.181324 (+ 801us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:44:09.182046 (+ 722us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:44:09.182182 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:09.182425 (+ 243us) server_negotiation.cc:299] Negotiation successful
0618 22:44:09.182533 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260618 22:44:09.199172 28630 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:09.196933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52926 (local address 127.5.213.252:41315)
0618 22:44:09.197116 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.197122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.197605 (+ 483us) server_negotiation.cc:407] Connection header received
0618 22:44:09.197823 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.197829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.197957 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.198070 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:09.198915 (+ 845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52926: BlockingRecv error: recv got EOF from 127.0.0.1:52926 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:44:09.211490 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:09.204567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49648 (local address 127.5.213.253:39699)
0618 22:44:09.204745 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.204750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.205288 (+ 538us) server_negotiation.cc:407] Connection header received
0618 22:44:09.205515 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.205520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.205617 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.205714 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:44:09.206184 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.207354 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.209375 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.210069 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.210197 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:09.210657 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:44:09.210824 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:44:09.210916 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:09.211076 (+ 160us) server_negotiation.cc:299] Negotiation successful
0618 22:44:09.211162 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260618 22:44:09.228824 28630 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:09.218475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.5.213.252:41315)
0618 22:44:09.218680 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.218687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.227286 (+ 8599us) server_negotiation.cc:407] Connection header received
0618 22:44:09.227518 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.227525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.227623 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.227721 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:09.228595 (+ 874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52942: BlockingRecv error: recv got EOF from 127.0.0.1:52942 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260618 22:44:09.241072 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:09.234083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49654 (local address 127.5.213.253:39699)
0618 22:44:09.234283 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.234289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.234803 (+ 514us) server_negotiation.cc:407] Connection header received
0618 22:44:09.234989 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.234995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.235092 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.235191 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:44:09.235705 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.236879 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.238913 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.239594 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.239743 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:09.240237 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:44:09.240400 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:44:09.240488 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:09.240637 (+ 149us) server_negotiation.cc:299] Negotiation successful
0618 22:44:09.240748 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260618 22:44:09.254252 28630 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:09.251094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.5.213.252:41315)
0618 22:44:09.251782 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.251788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.252684 (+ 896us) server_negotiation.cc:407] Connection header received
0618 22:44:09.252908 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.252915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.253010 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.253119 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:09.253994 (+ 875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52948: BlockingRecv error: recv got EOF from 127.0.0.1:52948 (error 108)
Metrics: {"server-negotiator.queue_time_us":515}
I20260618 22:44:09.271497 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:09.262425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49656 (local address 127.5.213.253:39699)
0618 22:44:09.262660 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.262666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.265150 (+ 2484us) server_negotiation.cc:407] Connection header received
0618 22:44:09.265388 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.265394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.265493 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.265599 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:44:09.266173 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.267413 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.269337 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.270019 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.270151 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:09.270709 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:44:09.270853 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:44:09.270945 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:09.271120 (+ 175us) server_negotiation.cc:299] Negotiation successful
0618 22:44:09.271206 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260618 22:44:09.286710 28630 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:09.284325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.5.213.252:41315)
0618 22:44:09.284585 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.284591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.285117 (+ 526us) server_negotiation.cc:407] Connection header received
0618 22:44:09.285323 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.285330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.285432 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.285550 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:09.286439 (+ 889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52950: BlockingRecv error: recv got EOF from 127.0.0.1:52950 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
I20260618 22:44:09.306600 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:09.299457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49666 (local address 127.5.213.253:39699)
0618 22:44:09.299657 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.299664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.300233 (+ 569us) server_negotiation.cc:407] Connection header received
0618 22:44:09.300439 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.300445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.300588 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.300725 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:44:09.301236 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.302405 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.304394 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.305104 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.305251 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:09.305701 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:44:09.305885 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:44:09.305982 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:09.306147 (+ 165us) server_negotiation.cc:299] Negotiation successful
0618 22:44:09.306265 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260618 22:44:09.319005 28630 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:09.316698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52952 (local address 127.5.213.252:41315)
0618 22:44:09.316881 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.316888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.317451 (+ 563us) server_negotiation.cc:407] Connection header received
0618 22:44:09.317645 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.317652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.317754 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.317889 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:09.318774 (+ 885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52952: BlockingRecv error: recv got EOF from 127.0.0.1:52952 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 22:44:09.345471 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:09.338665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.5.213.253:39699)
0618 22:44:09.338858 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.338864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.339441 (+ 577us) server_negotiation.cc:407] Connection header received
0618 22:44:09.339634 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.339639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.339745 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.339844 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:44:09.340354 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.341523 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.343476 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.344095 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.344241 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:09.344726 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:44:09.344877 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:44:09.344969 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:09.345102 (+ 133us) server_negotiation.cc:299] Negotiation successful
0618 22:44:09.345189 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260618 22:44:09.359030 28630 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:09.356459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52956 (local address 127.5.213.252:41315)
0618 22:44:09.356656 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.356661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.357241 (+ 580us) server_negotiation.cc:407] Connection header received
0618 22:44:09.357442 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.357448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.357547 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.357675 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:09.358805 (+ 1130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52956: BlockingRecv error: recv got EOF from 127.0.0.1:52956 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260618 22:44:09.400295 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:09.394685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.5.213.253:39699)
0618 22:44:09.394902 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.394907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.395184 (+ 277us) server_negotiation.cc:407] Connection header received
0618 22:44:09.395388 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.395392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.395475 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.395554 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:44:09.395983 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.396760 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.398583 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.399065 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.399153 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:09.399614 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:44:09.399715 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:44:09.399780 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:09.399921 (+ 141us) server_negotiation.cc:299] Negotiation successful
0618 22:44:09.399983 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260618 22:44:09.407670 28630 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:09.405909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52970 (local address 127.5.213.252:41315)
0618 22:44:09.406109 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.406115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.406369 (+ 254us) server_negotiation.cc:407] Connection header received
0618 22:44:09.406530 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.406536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.406660 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.406761 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:09.407412 (+ 651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52970: BlockingRecv error: recv got EOF from 127.0.0.1:52970 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260618 22:44:09.480890 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:09.474811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49696 (local address 127.5.213.253:39699)
0618 22:44:09.475032 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.475038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.475378 (+ 340us) server_negotiation.cc:407] Connection header received
0618 22:44:09.475576 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.475582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.475682 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.475808 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:44:09.476261 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.477420 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.478948 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.479606 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.479740 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:09.480137 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:44:09.480278 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:44:09.480375 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:09.480512 (+ 137us) server_negotiation.cc:299] Negotiation successful
0618 22:44:09.480602 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260618 22:44:09.501158 28630 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:09.499338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52986 (local address 127.5.213.252:41315)
0618 22:44:09.499535 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.499540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.499834 (+ 294us) server_negotiation.cc:407] Connection header received
0618 22:44:09.499997 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.500002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.500123 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.500238 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:09.500948 (+ 710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52986: BlockingRecv error: recv got EOF from 127.0.0.1:52986 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260618 22:44:09.637317 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:09.632271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49702 (local address 127.5.213.253:39699)
0618 22:44:09.632477 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.632483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.632768 (+ 285us) server_negotiation.cc:407] Connection header received
0618 22:44:09.632972 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.632976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.633044 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.633115 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:44:09.633532 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.634384 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.635681 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.636103 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.636222 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:09.636668 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:44:09.636783 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:44:09.636857 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:09.636989 (+ 132us) server_negotiation.cc:299] Negotiation successful
0618 22:44:09.637066 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260618 22:44:09.644047 28630 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:09.642464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53002 (local address 127.5.213.252:41315)
0618 22:44:09.642686 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.642691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.642928 (+ 237us) server_negotiation.cc:407] Connection header received
0618 22:44:09.643075 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.643080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.643148 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.643236 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:09.643864 (+ 628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53002: BlockingRecv error: recv got EOF from 127.0.0.1:53002 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 22:44:09.675443 28608 catalog_manager.cc:1259] Loaded cluster ID: 9a34685b90a84f4691e8b873593824b5
I20260618 22:44:09.675535 28608 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201: loading cluster ID for follower catalog manager: success
I20260618 22:44:09.677950 28608 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201: acquiring CA information for follower catalog manager: success
I20260618 22:44:09.679771 28608 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0e7fbff6d25f40928bf940da426b8201: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 22:44:09.907671 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:09.903297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59708 (local address 127.5.213.253:39699)
0618 22:44:09.903457 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.903461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.903704 (+ 243us) server_negotiation.cc:407] Connection header received
0618 22:44:09.903861 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.903866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.903949 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.904034 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:44:09.904359 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.905051 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.906345 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.906747 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.906835 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:09.907139 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:44:09.907255 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:44:09.907318 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:09.907399 (+ 81us) server_negotiation.cc:299] Negotiation successful
0618 22:44:09.907459 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 22:44:09.918460 28630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:09.912910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36266 (local address 127.5.213.252:41315)
0618 22:44:09.913060 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.913064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.913321 (+ 257us) server_negotiation.cc:407] Connection header received
0618 22:44:09.913475 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.913479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.913629 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.913736 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:44:09.914147 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.915091 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.916525 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.916958 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.917061 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:09.917525 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:44:09.918023 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:44:09.918148 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:09.918184 (+ 36us) server_negotiation.cc:299] Negotiation successful
0618 22:44:09.918254 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:44:09.934293 28773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:09.924294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34034 (local address 127.5.213.254:40785)
0618 22:44:09.928044 (+ 3750us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.928049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.928064 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:44:09.928155 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.928159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.928310 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.928497 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:44:09.929384 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.930471 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.932213 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.932643 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.932761 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:09.933270 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:44:09.933663 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:44:09.933764 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:09.933869 (+ 105us) server_negotiation.cc:299] Negotiation successful
0618 22:44:09.933959 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3648,"thread_start_us":3563,"threads_started":1}
I20260618 22:44:09.935483 28630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:09.929109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36282 (local address 127.5.213.252:41315)
0618 22:44:09.929289 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.929295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.929908 (+ 613us) server_negotiation.cc:407] Connection header received
0618 22:44:09.930054 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.930060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.930155 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.930251 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:44:09.930784 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.931481 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.933585 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.934202 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.934352 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:09.934829 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:44:09.934936 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:44:09.935012 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:09.935143 (+ 131us) server_negotiation.cc:299] Negotiation successful
0618 22:44:09.935222 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260618 22:44:09.936784 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:09.924336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.5.213.253:39699)
0618 22:44:09.924607 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:09.924615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:09.929096 (+ 4481us) server_negotiation.cc:407] Connection header received
0618 22:44:09.929394 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:09.929401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:09.929516 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:09.929600 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:44:09.930382 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.931416 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.934459 (+ 3043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:09.935044 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:09.935176 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:09.935950 (+ 774us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:44:09.936082 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:44:09.936167 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:09.936383 (+ 216us) server_negotiation.cc:299] Negotiation successful
0618 22:44:09.936469 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260618 22:44:09.939864 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 28376
I20260618 22:44:09.940832 28438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:44:10.052588 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 28376
I20260618 22:44:10.069763 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 28450
I20260618 22:44:10.070873 28512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:44:10.175241 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 28450
I20260618 22:44:10.190825 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 28524
I20260618 22:44:10.191928 28586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:44:10.296401 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 28524
2026-06-18T22:44:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4403 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:44:10 dist-test-slave-lnj3 krb5kdc[28783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:44:10 dist-test-slave-lnj3 krb5kdc[28783](info): set up 2 sockets
Jun 18 22:44:10 dist-test-slave-lnj3 krb5kdc[28783](info): commencing operation
krb5kdc: starting...
W20260618 22:44:11.315028 5975 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/test-user.keytab.
Jun 18 22:44:11 dist-test-slave-lnj3 krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:44:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:44:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
Entry for principal kudu/127.5.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab.
I20260618 22:44:11.561669 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:43177
--webserver_interface=127.5.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:36827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:43177,127.5.213.253:43849,127.5.213.252:41179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:44:11.892395 28796 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 22:44:11.892757 28796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:44:11.892866 28796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:44:11.892983 28796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:44:11.900482 28796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:44:11.900589 28796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:44:11.900672 28796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:44:11.900751 28796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:44:11.900840 28796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:44:11.911003 28796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:36827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:43177,127.5.213.253:43849,127.5.213.252:41179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.254.keytab
--principal=kudu/127.5.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.254:43177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:44:11.912276 28796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:44:11.913617 28796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:44:11.925114 28802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:44:11.925247 28804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:44:11.925277 28801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:44:11.926064 28796 server_base.cc:1034] running on GCE node
I20260618 22:44:11.926841 28796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:44:11.928170 28796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:44:11.929380 28796 hybrid_clock.cc:648] HybridClock initialized: now 1781822651929318 us; error 43 us; skew 500 ppm
I20260618 22:44:11.929689 28796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:44:11 dist-test-slave-lnj3 krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822651, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:44:11.937661 28796 init.cc:377] Logged in from keytab as kudu/127.5.213.254@KRBTEST.COM (short username kudu)
I20260618 22:44:11.939270 28796 webserver.cc:460] Webserver started at http://127.5.213.254:43123/ using document root <none> and password file <none>
I20260618 22:44:11.939997 28796 fs_manager.cc:359] Metadata directory not provided
I20260618 22:44:11.940124 28796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:44:11.940459 28796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:44:11.943655 28796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-0/data/instance:
uuid: "37d60df7d5aa49529da75283f6e93198"
format_stamp: "Formatted at 2026-06-18 22:44:11 on dist-test-slave-lnj3"
server_key: "418a9eb36b6420a22424240f08b3b8fb"
server_key_iv: "576f0ce9e4d76e6d2f1d463c48c1c872"
server_key_version: "encryptionkey@0"
I20260618 22:44:11.944466 28796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-0/wal/instance:
uuid: "37d60df7d5aa49529da75283f6e93198"
format_stamp: "Formatted at 2026-06-18 22:44:11 on dist-test-slave-lnj3"
server_key: "418a9eb36b6420a22424240f08b3b8fb"
server_key_iv: "576f0ce9e4d76e6d2f1d463c48c1c872"
server_key_version: "encryptionkey@0"
I20260618 22:44:11.949724 28796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260618 22:44:11.953132 28811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:44:11.954414 28796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:44:11.954582 28796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-0/wal
uuid: "37d60df7d5aa49529da75283f6e93198"
format_stamp: "Formatted at 2026-06-18 22:44:11 on dist-test-slave-lnj3"
server_key: "418a9eb36b6420a22424240f08b3b8fb"
server_key_iv: "576f0ce9e4d76e6d2f1d463c48c1c872"
server_key_version: "encryptionkey@0"
I20260618 22:44:11.954816 28796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:44:11.972541 28796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:44:11.976799 28796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:44:11.977080 28796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:44:11.997025 28796 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.254:43177
I20260618 22:44:11.997044 28862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.254:43177 every 8 connection(s)
I20260618 22:44:11.998428 28796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-0/data/info.pb
I20260618 22:44:12.002053 28863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:44:12.006067 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 28796
I20260618 22:44:12.006263 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-0/wal/instance
I20260618 22:44:12.006816 5975 external_mini_cluster.cc:1428] Setting key 6ba0b499414e0a880e0e0e25229992d1
I20260618 22:44:12.009472 28863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:44:12 dist-test-slave-lnj3 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822651, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.213.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.253@KRBTEST.COM" created.
I20260618 22:44:12.031635 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.014324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59040 (local address 127.5.213.254:43177)
0618 22:44:12.015024 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:12.015032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:12.015084 (+ 52us) server_negotiation.cc:407] Connection header received
0618 22:44:12.016365 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:12.016371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:12.016599 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:12.016779 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:12.018042 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.019250 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.020563 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.021111 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.021294 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:12.024745 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:12.024752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:12.024765 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:12.024806 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:12.027547 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:12.028155 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:12.028161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:12.028163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:12.028336 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:12.028677 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:12.028682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:12.028685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:12.029184 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:12.029375 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:12.029837 (+ 462us) server_negotiation.cc:299] Negotiation successful
0618 22:44:12.030166 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":504,"thread_start_us":356,"threads_started":1}
I20260618 22:44:12.031654 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.013020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43177 (local address 127.0.0.1:59040)
0618 22:44:12.013853 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.013906 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0618 22:44:12.015997 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:44:12.016815 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:44:12.016822 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:44:12.017515 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:44:12.017858 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:12.017875 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.019390 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.019394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:12.020437 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:12.020452 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.021324 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.021331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:12.021561 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:12.022213 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:44:12.022242 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:44:12.024545 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:44:12.027658 (+ 3113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:12.027669 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:12.027682 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:12.028045 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:12.028434 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:12.028439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:12.028442 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:12.028585 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:12.029304 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:44:12.029312 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:44:12.029509 (+ 197us) client_negotiation.cc:769] Sending connection context
0618 22:44:12.029722 (+ 213us) client_negotiation.cc:240] Negotiation successful
0618 22:44:12.030287 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":141,"thread_start_us":356,"threads_started":1}
I20260618 22:44:12.034484 28863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:44:12.035619 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.035183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:43849 (local address 127.0.0.1:58686)
0618 22:44:12.035379 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.035491 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260618 22:44:12.035913 28812 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.213.253:43849: connect: Connection refused (error 111)
I20260618 22:44:12.037442 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.037089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:43849 (local address 127.0.0.1:58698)
0618 22:44:12.037239 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.037305 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260618 22:44:12.037647 28863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.253:43849: Network error: Client connection negotiation failed: client connection to 127.5.213.253:43849: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.253.keytab.
Entry for principal kudu/127.5.213.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.253.keytab.
I20260618 22:44:12.057001 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.253:43849
--webserver_interface=127.5.213.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.253.keytab
--principal=kudu/127.5.213.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:36827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:43177,127.5.213.253:43849,127.5.213.252:41179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:44:12.086943 28863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } attempt: 1
I20260618 22:44:12.088065 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.087633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:43849 (local address 127.0.0.1:58706)
0618 22:44:12.087815 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.087936 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260618 22:44:12.089023 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.088671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:43849 (local address 127.0.0.1:58720)
0618 22:44:12.088829 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.088897 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260618 22:44:12.089200 28863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.253:43849: Network error: Client connection negotiation failed: client connection to 127.5.213.253:43849: connect: Connection refused (error 111)
I20260618 22:44:12.157455 28863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } attempt: 2
I20260618 22:44:12.158543 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.158070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:43849 (local address 127.0.0.1:58722)
0618 22:44:12.158310 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.158410 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260618 22:44:12.159595 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.159229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:43849 (local address 127.0.0.1:58724)
0618 22:44:12.159393 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.159469 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260618 22:44:12.159778 28863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.253:43849: Network error: Client connection negotiation failed: client connection to 127.5.213.253:43849: connect: Connection refused (error 111)
I20260618 22:44:12.250993 28863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } attempt: 3
I20260618 22:44:12.252139 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.251729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:43849 (local address 127.0.0.1:58738)
0618 22:44:12.251918 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.252008 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260618 22:44:12.253139 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.252801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:43849 (local address 127.0.0.1:58740)
0618 22:44:12.252935 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.253006 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260618 22:44:12.253371 28863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.253:43849: Network error: Client connection negotiation failed: client connection to 127.5.213.253:43849: connect: Connection refused (error 111)
W20260618 22:44:12.369059 28870 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 22:44:12.369382 28870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:44:12.369488 28870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:44:12.369597 28870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:44:12.376914 28870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:44:12.377012 28870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:44:12.377076 28870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:44:12.377146 28870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:44:12.377235 28870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:44:12.387281 28870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:36827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:43177,127.5.213.253:43849,127.5.213.252:41179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.253.keytab
--principal=kudu/127.5.213.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.253:43849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.213.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:44:12.388561 28870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:44:12.389830 28870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:44:12.396588 28863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } attempt: 4
I20260618 22:44:12.397698 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.397213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:43849 (local address 127.0.0.1:58748)
0618 22:44:12.397486 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.397570 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260618 22:44:12.398586 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.398289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:43849 (local address 127.0.0.1:58762)
0618 22:44:12.398402 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.398469 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.253:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260618 22:44:12.398841 28863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.253:43849: Network error: Client connection negotiation failed: client connection to 127.5.213.253:43849: connect: Connection refused (error 111)
W20260618 22:44:12.401424 28876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:44:12.401582 28875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:44:12.401556 28878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:44:12.402444 28870 server_base.cc:1034] running on GCE node
I20260618 22:44:12.403247 28870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:44:12.404603 28870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:44:12.405800 28870 hybrid_clock.cc:648] HybridClock initialized: now 1781822652405735 us; error 40 us; skew 500 ppm
I20260618 22:44:12.406105 28870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:44:12 dist-test-slave-lnj3 krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822652, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:44:12.409098 28870 init.cc:377] Logged in from keytab as kudu/127.5.213.253@KRBTEST.COM (short username kudu)
I20260618 22:44:12.410514 28870 webserver.cc:460] Webserver started at http://127.5.213.253:44171/ using document root <none> and password file <none>
I20260618 22:44:12.411196 28870 fs_manager.cc:359] Metadata directory not provided
I20260618 22:44:12.411291 28870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:44:12.411549 28870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:44:12.414366 28870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-1/data/instance:
uuid: "fef366de892b4719914e1c618f6ae0c2"
format_stamp: "Formatted at 2026-06-18 22:44:12 on dist-test-slave-lnj3"
server_key: "b82f393424c136fd53ceb453fa1bd9e0"
server_key_iv: "794a744fa2cbeabe0f744f31e85bb9fa"
server_key_version: "encryptionkey@0"
I20260618 22:44:12.415150 28870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-1/wal/instance:
uuid: "fef366de892b4719914e1c618f6ae0c2"
format_stamp: "Formatted at 2026-06-18 22:44:12 on dist-test-slave-lnj3"
server_key: "b82f393424c136fd53ceb453fa1bd9e0"
server_key_iv: "794a744fa2cbeabe0f744f31e85bb9fa"
server_key_version: "encryptionkey@0"
I20260618 22:44:12.420243 28870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260618 22:44:12.423571 28885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:44:12.424877 28870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:44:12.425024 28870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-1/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-1/wal
uuid: "fef366de892b4719914e1c618f6ae0c2"
format_stamp: "Formatted at 2026-06-18 22:44:12 on dist-test-slave-lnj3"
server_key: "b82f393424c136fd53ceb453fa1bd9e0"
server_key_iv: "794a744fa2cbeabe0f744f31e85bb9fa"
server_key_version: "encryptionkey@0"
I20260618 22:44:12.425230 28870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:44:12.450193 28870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:44:12.454486 28870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:44:12.454766 28870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:44:12.471335 28870 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.253:43849
I20260618 22:44:12.471359 28936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.253:43849 every 8 connection(s)
I20260618 22:44:12.472648 28870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-1/data/info.pb
I20260618 22:44:12.476234 28937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:44:12.478744 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 28870
I20260618 22:44:12.478932 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-1/wal/instance
I20260618 22:44:12.479454 5975 external_mini_cluster.cc:1428] Setting key 9205131e0eeb1cd779e49e79d031f3ca
I20260618 22:44:12.484329 28937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:44:12 dist-test-slave-lnj3 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822652, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.253@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:44:12.505970 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.487380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.5.213.254:43177)
0618 22:44:12.487581 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:12.487587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:12.488821 (+ 1234us) server_negotiation.cc:407] Connection header received
0618 22:44:12.490884 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:12.490889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:12.490959 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:12.491048 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:12.492952 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.493723 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.495789 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.496201 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.496312 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:12.500181 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:12.500186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:12.500190 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:12.500225 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:12.502817 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:12.503729 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:12.503733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:12.503735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:12.503853 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:12.504300 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:12.504302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:12.504304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:12.504579 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:12.504682 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:12.505574 (+ 892us) server_negotiation.cc:299] Negotiation successful
0618 22:44:12.505746 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.5.213.252@KRBTEST.COM; defaulting to no policy
I20260618 22:44:12.507285 28939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.487866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43177 (local address 127.0.0.1:59054)
0618 22:44:12.488647 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.488698 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 22:44:12.490556 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:44:12.491305 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:44:12.491317 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:44:12.491980 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:44:12.492743 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:12.492766 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.493903 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.493909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:12.495659 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:12.495675 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.496413 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.496420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:12.496694 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:12.497358 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:44:12.497387 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:44:12.499933 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:44:12.502968 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:12.502978 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:12.502998 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:12.503623 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:12.503949 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:12.503955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:12.503958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:12.504201 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:12.504698 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:44:12.504707 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:44:12.505187 (+ 480us) client_negotiation.cc:769] Sending connection context
0618 22:44:12.505508 (+ 321us) client_negotiation.cc:240] Negotiation successful
0618 22:44:12.506038 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":331,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.213.252@KRBTEST.COM" created.
I20260618 22:44:12.508973 28937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:44:12 dist-test-slave-lnj3 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822652, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.253@KRBTEST.COM for kudu/127.5.213.253@KRBTEST.COM
I20260618 22:44:12.523389 28939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.509707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:43849 (local address 127.0.0.1:58774)
0618 22:44:12.509870 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.509886 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:44:12.509984 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:44:12.510931 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:44:12.510936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:44:12.511375 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:44:12.511541 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:12.511550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.512841 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.512846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:12.514043 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:12.514055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.514903 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.514909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:12.515072 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:12.515603 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:44:12.515624 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:44:12.517905 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:44:12.521020 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:12.521025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:12.521028 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:12.521470 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:12.521792 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:12.521797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:12.521800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:12.521949 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:12.522494 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:44:12.522498 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:44:12.522654 (+ 156us) client_negotiation.cc:769] Sending connection context
0618 22:44:12.522780 (+ 126us) client_negotiation.cc:240] Negotiation successful
0618 22:44:12.523048 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260618 22:44:12.523423 28941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.509908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.5.213.253:43849)
0618 22:44:12.510488 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:12.510494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:12.510518 (+ 24us) server_negotiation.cc:407] Connection header received
0618 22:44:12.510560 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:12.510566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:12.510789 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:12.510901 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:12.511686 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.512707 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.514163 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.514745 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.514839 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:12.518111 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:12.518125 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:12.518142 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:12.518183 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:12.520903 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:12.521581 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:12.521588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:12.521590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:12.521698 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:12.522047 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:12.522050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:12.522052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:12.522371 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:12.522518 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:12.522916 (+ 398us) server_negotiation.cc:299] Negotiation successful
0618 22:44:12.523139 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":444,"thread_start_us":315,"threads_started":1}
I20260618 22:44:12.525862 28937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:44:12.526993 28939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.526537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37710)
0618 22:44:12.526722 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.526855 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260618 22:44:12.527244 28888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
I20260618 22:44:12.528704 28939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.528373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37722)
0618 22:44:12.528509 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.528582 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260618 22:44:12.528916 28937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:41179: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.213.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.252.keytab.
Entry for principal kudu/127.5.213.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.252.keytab.
I20260618 22:44:12.539601 5975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
/tmp/dist-test-taskcnPxcP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.252:41179
--webserver_interface=127.5.213.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.252.keytab
--principal=kudu/127.5.213.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.213.212:36827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.213.254:43177,127.5.213.253:43849,127.5.213.252:41179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:44:12.578210 28937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } attempt: 1
I20260618 22:44:12.579361 28939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.578915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37724)
0618 22:44:12.579100 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.579204 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260618 22:44:12.580272 28939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.579938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37726)
0618 22:44:12.580096 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.580168 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260618 22:44:12.580466 28937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:41179: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
I20260618 22:44:12.648656 28937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } attempt: 2
I20260618 22:44:12.649708 28939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.649280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37730)
0618 22:44:12.649467 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.649557 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260618 22:44:12.650592 28939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.650282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37742)
0618 22:44:12.650413 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.650480 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260618 22:44:12.650859 28937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:41179: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
I20260618 22:44:12.698055 28863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } attempt: 5
Jun 18 22:44:12 dist-test-slave-lnj3 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822651, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for kudu/127.5.213.253@KRBTEST.COM
I20260618 22:44:12.711789 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.698800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:43849 (local address 127.0.0.1:58786)
0618 22:44:12.699007 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.699026 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:44:12.699159 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:44:12.699524 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:44:12.699527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:44:12.699892 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:44:12.700067 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:12.700080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.701437 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.701440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:12.702265 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:12.702275 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.703239 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.703243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:12.703378 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:12.703917 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:44:12.703934 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:44:12.706054 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:44:12.709673 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:12.709677 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:12.709680 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:12.710027 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:12.710401 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:12.710404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:12.710406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:12.710522 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:12.711104 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:44:12.711107 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:44:12.711232 (+ 125us) client_negotiation.cc:769] Sending connection context
0618 22:44:12.711340 (+ 108us) client_negotiation.cc:240] Negotiation successful
0618 22:44:12.711543 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260618 22:44:12.711989 28941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.698873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58786 (local address 127.5.213.253:43849)
0618 22:44:12.699017 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:12.699023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:12.699131 (+ 108us) server_negotiation.cc:407] Connection header received
0618 22:44:12.699294 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:12.699300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:12.699385 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:12.699493 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:12.700221 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.701326 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.702412 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.703117 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.703239 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:12.706255 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:12.706261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:12.706266 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:12.706309 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:12.709547 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:12.710141 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:12.710147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:12.710151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:12.710304 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:12.710614 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:12.710619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:12.710622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:12.710998 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:12.711143 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:12.711466 (+ 323us) server_negotiation.cc:299] Negotiation successful
0618 22:44:12.711691 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:44:12.712914 28863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:44:12.713886 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.713541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37746)
0618 22:44:12.713691 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.713759 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260618 22:44:12.714818 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.714486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37748)
0618 22:44:12.714601 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.714705 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260618 22:44:12.714995 28863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:41179: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
I20260618 22:44:12.742057 28937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } attempt: 3
I20260618 22:44:12.743168 28939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.742711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37750)
0618 22:44:12.742929 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.743016 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260618 22:44:12.744272 28939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.743872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37754)
0618 22:44:12.744092 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.744166 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260618 22:44:12.744499 28937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:41179: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
I20260618 22:44:12.766186 28863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } attempt: 1
I20260618 22:44:12.767302 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.766847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37768)
0618 22:44:12.767026 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.767128 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260618 22:44:12.768180 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.767873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37780)
0618 22:44:12.768011 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.768079 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260618 22:44:12.768363 28863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:41179: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
I20260618 22:44:12.836552 28863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } attempt: 2
I20260618 22:44:12.837581 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.837112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37794)
0618 22:44:12.837342 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.837429 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260618 22:44:12.838475 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.838161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37808)
0618 22:44:12.838301 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.838376 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260618 22:44:12.838685 28863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:41179: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
W20260618 22:44:12.859508 28944 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 22:44:12.859853 28944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:44:12.859951 28944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:44:12.860061 28944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:44:12.867324 28944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:44:12.867417 28944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:44:12.867498 28944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:44:12.867559 28944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:44:12.867653 28944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:44:12.877120 28944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.213.212:36827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.213.254:43177,127.5.213.253:43849,127.5.213.252:41179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/krb5kdc/kudu_127.5.213.252.keytab
--principal=kudu/127.5.213.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.213.252:41179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.213.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 21:43:06 UTC on ccycloud.quasar-uighkb.root.comops.site
build id 4454736
ASAN enabled
I20260618 22:44:12.878324 28944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:44:12.879619 28944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:44:12.887679 28937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } attempt: 4
I20260618 22:44:12.888765 28939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.888295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37822)
0618 22:44:12.888490 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.888589 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260618 22:44:12.890045 28939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.889600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37832)
0618 22:44:12.889831 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.889917 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260618 22:44:12.890367 28937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:41179: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
W20260618 22:44:12.890311 28950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:44:12.890750 28949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:44:12.890821 28952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:44:12.891443 28944 server_base.cc:1034] running on GCE node
I20260618 22:44:12.892215 28944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:44:12.893551 28944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:44:12.894742 28944 hybrid_clock.cc:648] HybridClock initialized: now 1781822652894672 us; error 43 us; skew 500 ppm
I20260618 22:44:12.895059 28944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 18 22:44:12 dist-test-slave-lnj3 krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822652, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:44:12.898223 28944 init.cc:377] Logged in from keytab as kudu/127.5.213.252@KRBTEST.COM (short username kudu)
I20260618 22:44:12.899750 28944 webserver.cc:460] Webserver started at http://127.5.213.252:39411/ using document root <none> and password file <none>
I20260618 22:44:12.900452 28944 fs_manager.cc:359] Metadata directory not provided
I20260618 22:44:12.900556 28944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:44:12.900906 28944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:44:12.904072 28944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-2/data/instance:
uuid: "920e14c4d60141c89a4376089515c92d"
format_stamp: "Formatted at 2026-06-18 22:44:12 on dist-test-slave-lnj3"
server_key: "8e0d614bc4017e459954093c4246ade1"
server_key_iv: "cb824dbe57971395676a2cbf800ebfaf"
server_key_version: "encryptionkey@0"
I20260618 22:44:12.904896 28944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-2/wal/instance:
uuid: "920e14c4d60141c89a4376089515c92d"
format_stamp: "Formatted at 2026-06-18 22:44:12 on dist-test-slave-lnj3"
server_key: "8e0d614bc4017e459954093c4246ade1"
server_key_iv: "cb824dbe57971395676a2cbf800ebfaf"
server_key_version: "encryptionkey@0"
I20260618 22:44:12.910189 28944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260618 22:44:12.913575 28959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:44:12.915021 28944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:44:12.915169 28944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-2/data,/tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-2/wal
uuid: "920e14c4d60141c89a4376089515c92d"
format_stamp: "Formatted at 2026-06-18 22:44:12 on dist-test-slave-lnj3"
server_key: "8e0d614bc4017e459954093c4246ade1"
server_key_iv: "cb824dbe57971395676a2cbf800ebfaf"
server_key_version: "encryptionkey@0"
I20260618 22:44:12.915356 28944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:44:12.927255 28944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:44:12.931032 28944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:44:12.931267 28944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:44:12.944888 28863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } attempt: 3
I20260618 22:44:12.946045 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.945496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37844)
0618 22:44:12.945736 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.945846 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260618 22:44:12.947036 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.946667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37854)
0618 22:44:12.946853 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.946931 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260618 22:44:12.947254 28863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.213.252:41179: Network error: Client connection negotiation failed: client connection to 127.5.213.252:41179: connect: Connection refused (error 111)
I20260618 22:44:12.948078 28944 rpc_server.cc:307] RPC server started. Bound to: 127.5.213.252:41179
I20260618 22:44:12.948097 29010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.213.252:41179 every 8 connection(s)
I20260618 22:44:12.949375 28944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-2/data/info.pb
I20260618 22:44:12.951761 5975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu as pid 28944
I20260618 22:44:12.951916 5975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnPxcP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781822307836422-5975-0/minicluster-data/master-2/wal/instance
I20260618 22:44:12.952315 5975 external_mini_cluster.cc:1428] Setting key a4274b61ee2b546fb37e2316686c87cb
I20260618 22:44:12.953086 29011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:44:12.962456 29011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:44:12 dist-test-slave-lnj3 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.252@KRBTEST.COM
Jun 18 22:44:12 dist-test-slave-lnj3 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822652, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.252@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
I20260618 22:44:12.975507 29014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.953958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37864 (local address 127.5.213.252:41179)
0618 22:44:12.954699 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:12.954714 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:12.954763 (+ 49us) server_negotiation.cc:407] Connection header received
0618 22:44:12.956406 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:12.956433 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:12.956978 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:12.957411 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:12.958760 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.960836 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.962052 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.963069 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.963257 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:12.967786 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:12.967798 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:12.967813 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:12.967862 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:12.971143 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:12.971976 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:12.971983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:12.971989 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:12.972178 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:12.972615 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:12.972619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:12.972622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:12.973170 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:12.973455 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:12.973949 (+ 494us) server_negotiation.cc:299] Negotiation successful
0618 22:44:12.974369 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":301,"threads_started":1}
I20260618 22:44:12.980748 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.965298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.254:43177 (local address 127.0.0.1:59056)
0618 22:44:12.965929 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.965974 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 22:44:12.966119 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:44:12.966512 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:44:12.966519 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:44:12.967309 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:44:12.967609 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:12.967621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.968871 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.968875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:12.970109 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:12.970122 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.971079 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.971083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:12.971206 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:12.971768 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:44:12.971791 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:44:12.974133 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:44:12.977820 (+ 3687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:12.977831 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:12.977838 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:12.978356 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:12.978975 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:12.978981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:12.978985 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:12.979161 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:12.979825 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:44:12.979831 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:44:12.979997 (+ 166us) client_negotiation.cc:769] Sending connection context
0618 22:44:12.980200 (+ 203us) client_negotiation.cc:240] Negotiation successful
0618 22:44:12.980445 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":338,"threads_started":1}
I20260618 22:44:12.980934 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.965168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59056 (local address 127.5.213.254:43177)
0618 22:44:12.965313 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:12.965317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:12.966068 (+ 751us) server_negotiation.cc:407] Connection header received
0618 22:44:12.966230 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:12.966236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:12.966345 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:12.966425 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:12.967750 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.968720 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.970253 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.970916 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.971050 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:12.974341 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:12.974346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:12.974350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:12.974384 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:12.977696 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:12.978491 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:12.978497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:12.978500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:12.978785 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:12.979282 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:12.979286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:12.979288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:12.979670 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:12.979810 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:12.980340 (+ 530us) server_negotiation.cc:299] Negotiation successful
0618 22:44:12.980557 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:44:12.982228 29011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:44:12 dist-test-slave-lnj3 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.254@KRBTEST.COM
Jun 18 22:44:12 dist-test-slave-lnj3 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822652, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.252@KRBTEST.COM for kudu/127.5.213.253@KRBTEST.COM
I20260618 22:44:12.994086 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.978533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59058 (local address 127.5.213.254:43177)
0618 22:44:12.979169 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:12.979174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:12.979199 (+ 25us) server_negotiation.cc:407] Connection header received
0618 22:44:12.979250 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:12.979256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:12.979329 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:12.979416 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:12.980950 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.981939 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.983530 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.983976 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.984064 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:12.988422 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:12.988427 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:12.988430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:12.988471 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:12.991589 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:12.992222 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:12.992227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:12.992231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:12.992365 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:12.992705 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:12.992710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:12.992713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:12.992972 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:12.993251 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:12.993523 (+ 272us) server_negotiation.cc:299] Negotiation successful
0618 22:44:12.993740 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":258,"threads_started":1}
I20260618 22:44:12.995038 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.982906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.253:43849 (local address 127.0.0.1:58794)
0618 22:44:12.983098 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.983118 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:44:12.983230 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:44:12.983518 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:44:12.983524 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:44:12.983877 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:44:12.984033 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:12.984045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.985096 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.985101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:12.986490 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:12.986505 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.987192 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.987197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:12.987345 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:12.987869 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:44:12.987886 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:44:12.989810 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:44:12.992847 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:12.992852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:12.992856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:12.993193 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:12.993551 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:12.993554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:12.993556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:12.993683 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:12.994213 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:44:12.994217 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:44:12.994384 (+ 167us) client_negotiation.cc:769] Sending connection context
0618 22:44:12.994535 (+ 151us) client_negotiation.cc:240] Negotiation successful
0618 22:44:12.994812 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260618 22:44:12.995483 28941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.982950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58794 (local address 127.5.213.253:43849)
0618 22:44:12.983116 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:12.983120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:12.983168 (+ 48us) server_negotiation.cc:407] Connection header received
0618 22:44:12.983335 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:12.983339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:12.983394 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:12.983465 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:12.984197 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.984958 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.986614 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.987056 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.987175 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:12.990006 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:12.990012 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:12.990017 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:12.990058 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:12.992725 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:12.993320 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:12.993325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:12.993327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:12.993450 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:12.993779 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:12.993784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:12.993787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:12.994095 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:12.994229 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:12.994677 (+ 448us) server_negotiation.cc:299] Negotiation successful
0618 22:44:12.994899 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260618 22:44:12.996290 29011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:44:12 dist-test-slave-lnj3 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822652, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.252@KRBTEST.COM for kudu/127.5.213.252@KRBTEST.COM
Jun 18 22:44:13 dist-test-slave-lnj3 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.213.253@KRBTEST.COM
I20260618 22:44:13.009406 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.996931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37876)
0618 22:44:12.997207 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:12.997222 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:44:12.997313 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:44:12.997559 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:44:12.997563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:44:12.997924 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:44:12.998046 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:12.998054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.999001 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.999007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:12.999848 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:12.999859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:13.000515 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:13.000521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:13.000690 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:13.001220 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:44:13.001235 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:44:13.003648 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:44:13.007096 (+ 3448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:13.007099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:13.007102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:13.007476 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:13.007847 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:13.007850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:13.007852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:13.007968 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:13.008505 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:44:13.008508 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:44:13.008624 (+ 116us) client_negotiation.cc:769] Sending connection context
0618 22:44:13.008747 (+ 123us) client_negotiation.cc:240] Negotiation successful
0618 22:44:13.009021 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260618 22:44:13.009541 29014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.997084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37876 (local address 127.5.213.252:41179)
0618 22:44:12.997210 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:12.997215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:12.997263 (+ 48us) server_negotiation.cc:407] Connection header received
0618 22:44:12.997404 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:12.997409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:12.997465 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:12.997534 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:12.998160 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:12.998898 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:12.999962 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:13.000381 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:13.000479 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:13.003847 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:13.003853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:13.003857 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:13.003892 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:13.006979 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:13.007584 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:13.007589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:13.007593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:13.007739 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:13.008060 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:13.008065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:13.008068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:13.008409 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:13.008550 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:13.008868 (+ 318us) server_negotiation.cc:299] Negotiation successful
0618 22:44:13.009199 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":28}
I20260618 22:44:13.014510 29011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d: Bootstrap starting.
I20260618 22:44:13.015672 28941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:12.996017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58806 (local address 127.5.213.253:43849)
0618 22:44:12.996164 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:13.001176 (+ 5012us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:13.001205 (+ 29us) server_negotiation.cc:407] Connection header received
0618 22:44:13.001265 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:13.001270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:13.001342 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:13.001429 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:13.002907 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:13.003626 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:13.005149 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:13.005558 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:13.005649 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:13.010261 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:13.010267 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:13.010271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:13.010310 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:13.012890 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:13.013711 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:13.013731 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:13.013733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:13.013881 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:13.014278 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:13.014281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:13.014283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:13.014544 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:13.014865 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:13.015146 (+ 281us) server_negotiation.cc:299] Negotiation successful
0618 22:44:13.015373 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:44:13.018208 29011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d: Neither blocks nor log segments found. Creating new log.
I20260618 22:44:13.019441 29011 log.cc:824] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d: Log is configured to *not* fsync() on all Append() calls
I20260618 22:44:13.020404 5975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:44:13.022303 29011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d: No bootstrap required, opened a new log
I20260618 22:44:13.022747 28838 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "37d60df7d5aa49529da75283f6e93198"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59058
I20260618 22:44:13.032845 29011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d60df7d5aa49529da75283f6e93198" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } } peers { permanent_uuid: "fef366de892b4719914e1c618f6ae0c2" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } } peers { permanent_uuid: "920e14c4d60141c89a4376089515c92d" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } }
I20260618 22:44:13.033260 29011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 920e14c4d60141c89a4376089515c92d, State: Initialized, Role: FOLLOWER
I20260618 22:44:13.033761 29011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d60df7d5aa49529da75283f6e93198" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } } peers { permanent_uuid: "fef366de892b4719914e1c618f6ae0c2" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } } peers { permanent_uuid: "920e14c4d60141c89a4376089515c92d" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } }
I20260618 22:44:13.034781 29018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d60df7d5aa49529da75283f6e93198" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } } peers { permanent_uuid: "fef366de892b4719914e1c618f6ae0c2" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } } peers { permanent_uuid: "920e14c4d60141c89a4376089515c92d" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } } }
I20260618 22:44:13.035087 29018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:44:13.036010 29011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d [sys.catalog]: configured and running, proceeding with master startup.
W20260618 22:44:13.044960 29029 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:44:13.045091 29029 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:44:13.042794 28944 master_runner.cc:186] Error getting master registration for 127.5.213.254:43177: OK, instance_id { permanent_uuid: "37d60df7d5aa49529da75283f6e93198" instance_seqno: 1781822651991833 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 22:44:13.053833 28944 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 22:44:13.124428 28863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } attempt: 4
Jun 18 22:44:13 dist-test-slave-lnj3 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822651, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.254@KRBTEST.COM for kudu/127.5.213.252@KRBTEST.COM
I20260618 22:44:13.139240 29014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:13.125233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37878 (local address 127.5.213.252:41179)
0618 22:44:13.125427 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:13.125432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:13.125452 (+ 20us) server_negotiation.cc:407] Connection header received
0618 22:44:13.125657 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:13.125663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:13.125763 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:13.125872 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:13.126840 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:13.127967 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:13.129309 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:13.130050 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:13.130171 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:13.133208 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:13.133214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:13.133217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:13.133248 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:13.136478 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:13.137079 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:13.137085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:13.137089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:13.137247 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:13.137673 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:13.137678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:13.137681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:13.138033 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:13.138197 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:13.138607 (+ 410us) server_negotiation.cc:299] Negotiation successful
0618 22:44:13.138863 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260618 22:44:13.139248 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:13.125145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37878)
0618 22:44:13.125349 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:13.125370 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:44:13.125504 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:44:13.125912 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:44:13.125917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:44:13.126470 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:44:13.126687 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:13.126699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:13.128090 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:13.128096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:13.129151 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:13.129167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:13.130203 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:13.130208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:13.130419 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:13.130990 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:44:13.131008 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:44:13.133027 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:44:13.136613 (+ 3586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:13.136618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:13.136623 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:13.136970 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:13.137367 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:13.137372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:13.137376 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:13.137576 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:13.138158 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:44:13.138163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:44:13.138356 (+ 193us) client_negotiation.cc:769] Sending connection context
0618 22:44:13.138508 (+ 152us) client_negotiation.cc:240] Negotiation successful
0618 22:44:13.138827 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260618 22:44:13.143611 28863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198: Bootstrap starting.
I20260618 22:44:13.146908 28863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198: Neither blocks nor log segments found. Creating new log.
I20260618 22:44:13.147933 28863 log.cc:824] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198: Log is configured to *not* fsync() on all Append() calls
I20260618 22:44:13.150694 28863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198: No bootstrap required, opened a new log
I20260618 22:44:13.155700 28863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d60df7d5aa49529da75283f6e93198" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } } peers { permanent_uuid: "fef366de892b4719914e1c618f6ae0c2" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } } peers { permanent_uuid: "920e14c4d60141c89a4376089515c92d" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } }
I20260618 22:44:13.156000 28863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37d60df7d5aa49529da75283f6e93198, State: Initialized, Role: FOLLOWER
I20260618 22:44:13.156432 28863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d60df7d5aa49529da75283f6e93198" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } } peers { permanent_uuid: "fef366de892b4719914e1c618f6ae0c2" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } } peers { permanent_uuid: "920e14c4d60141c89a4376089515c92d" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } }
I20260618 22:44:13.157274 29032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d60df7d5aa49529da75283f6e93198" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } } peers { permanent_uuid: "fef366de892b4719914e1c618f6ae0c2" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } } peers { permanent_uuid: "920e14c4d60141c89a4376089515c92d" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } } }
I20260618 22:44:13.157639 29032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:44:13.158491 28863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:44:13.158583 28838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260618 22:44:13.158876 28838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:44:13.162075 28838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d60df7d5aa49529da75283f6e93198" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } } peers { permanent_uuid: "fef366de892b4719914e1c618f6ae0c2" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } } peers { permanent_uuid: "920e14c4d60141c89a4376089515c92d" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } }
I20260618 22:44:13.164367 28838 leader_election.cc:288] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [CANDIDATE]: Term 1 election: Requested vote from peers fef366de892b4719914e1c618f6ae0c2 (127.5.213.253:43849), 920e14c4d60141c89a4376089515c92d (127.5.213.252:41179)
I20260618 22:44:13.164789 28986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "37d60df7d5aa49529da75283f6e93198" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "920e14c4d60141c89a4376089515c92d"
I20260618 22:44:13.165253 28986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:44:13.168551 28986 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37d60df7d5aa49529da75283f6e93198 in term 1.
I20260618 22:44:13.169217 28814 leader_election.cc:302] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37d60df7d5aa49529da75283f6e93198, 920e14c4d60141c89a4376089515c92d; no voters:
I20260618 22:44:13.169703 29032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:44:13.170738 28796 master_runner.cc:186] Error getting master registration for 127.5.213.253:43849: OK, instance_id { permanent_uuid: "fef366de892b4719914e1c618f6ae0c2" instance_seqno: 1781822652466386 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 22:44:13.182509 28796 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260618 22:44:13.183131 28965 tablet.cc:2369] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:44:13.189566 28937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } attempt: 5
I20260618 22:44:13.199631 29032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [term 1 LEADER]: Becoming Leader. State: Replica: 37d60df7d5aa49529da75283f6e93198, State: Running, Role: LEADER
I20260618 22:44:13.200368 29032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d60df7d5aa49529da75283f6e93198" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } } peers { permanent_uuid: "fef366de892b4719914e1c618f6ae0c2" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } } peers { permanent_uuid: "920e14c4d60141c89a4376089515c92d" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } }
W20260618 22:44:13.202909 29044 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:44:13.203090 29044 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:44:13.212726 29045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37d60df7d5aa49529da75283f6e93198. Latest consensus state: current_term: 1 leader_uuid: "37d60df7d5aa49529da75283f6e93198" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d60df7d5aa49529da75283f6e93198" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } } peers { permanent_uuid: "fef366de892b4719914e1c618f6ae0c2" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } } peers { permanent_uuid: "920e14c4d60141c89a4376089515c92d" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } } }
I20260618 22:44:13.213135 29045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [sys.catalog]: This master's current role is: LEADER
Jun 18 22:44:13 dist-test-slave-lnj3 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781822652, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.213.253@KRBTEST.COM for kudu/127.5.213.252@KRBTEST.COM
I20260618 22:44:13.223263 28939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:13.190188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.213.252:41179 (local address 127.0.0.1:37906)
0618 22:44:13.190447 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0618 22:44:13.190468 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:44:13.190587 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:44:13.205394 (+ 14807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:44:13.205400 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:44:13.205901 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:44:13.206089 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:13.206101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:13.208631 (+ 2530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:13.208650 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:13.210111 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:44:13.210135 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:44:13.211615 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:44:13.211622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:44:13.211857 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:13.212598 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:44:13.212624 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:44:13.215682 (+ 3058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:44:13.220311 (+ 4629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:13.220318 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:13.220322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:13.220881 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:13.221405 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:44:13.221411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:44:13.221415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:44:13.221588 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:44:13.222227 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:44:13.222232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:44:13.222395 (+ 163us) client_negotiation.cc:769] Sending connection context
0618 22:44:13.222547 (+ 152us) client_negotiation.cc:240] Negotiation successful
0618 22:44:13.222873 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260618 22:44:13.223574 29053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:13.190905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37906 (local address 127.5.213.252:41179)
0618 22:44:13.204999 (+ 14094us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:13.205006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:13.205048 (+ 42us) server_negotiation.cc:407] Connection header received
0618 22:44:13.205115 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:13.205121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:13.205233 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:13.205347 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:13.206329 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:13.208345 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:13.210416 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:13.211460 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:13.211586 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:13.215955 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:13.215962 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:13.215969 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:13.216013 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:13.220143 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:13.221052 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:13.221058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:13.221063 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:13.221258 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:13.221705 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:13.221710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:13.221714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:13.222090 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:13.222240 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:13.222685 (+ 445us) server_negotiation.cc:299] Negotiation successful
0618 22:44:13.222928 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13912,"thread_start_us":13688,"threads_started":1}
I20260618 22:44:13.229941 28937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fef366de892b4719914e1c618f6ae0c2: Bootstrap starting.
W20260618 22:44:13.230943 28817 tablet.cc:2369] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:44:13.234488 28937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fef366de892b4719914e1c618f6ae0c2: Neither blocks nor log segments found. Creating new log.
I20260618 22:44:13.235968 28937 log.cc:824] T 00000000000000000000000000000000 P fef366de892b4719914e1c618f6ae0c2: Log is configured to *not* fsync() on all Append() calls
I20260618 22:44:13.238118 28941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:13.188091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.5.213.253:43849)
0618 22:44:13.188306 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:13.188313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:13.192443 (+ 4130us) server_negotiation.cc:407] Connection header received
0618 22:44:13.192765 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:13.192772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:13.192857 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:13.192928 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:13.203069 (+ 10141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:13.204250 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:13.218550 (+ 14300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:13.219247 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:13.219343 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:13.229471 (+ 10128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:13.229478 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:13.229482 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:13.229521 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:13.233173 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:13.235676 (+ 2503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:13.235683 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:13.235687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:13.235864 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:13.236339 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:13.236345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:13.236348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:13.236723 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:13.236967 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:13.237519 (+ 552us) server_negotiation.cc:299] Negotiation successful
0618 22:44:13.237769 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260618 22:44:13.242435 29055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:44:13.248040 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:13.188132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59074 (local address 127.5.213.254:43177)
0618 22:44:13.188337 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:13.188343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:13.199211 (+ 10868us) server_negotiation.cc:407] Connection header received
0618 22:44:13.199402 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:13.199408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:13.199538 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:13.199664 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:13.205370 (+ 5706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:13.206434 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:13.211822 (+ 5388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:13.212535 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:13.212654 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:13.218307 (+ 5653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:13.218314 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:13.218324 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:13.218369 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:13.234494 (+ 16125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:13.242620 (+ 8126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:13.242641 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:13.242645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:13.242825 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:13.244524 (+ 1699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:13.244529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:13.244532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:13.244900 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:13.245144 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:13.247452 (+ 2308us) server_negotiation.cc:299] Negotiation successful
0618 22:44:13.247701 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260618 22:44:13.253002 29014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:13.189233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37890 (local address 127.5.213.252:41179)
0618 22:44:13.189411 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:13.189417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:13.229322 (+ 39905us) server_negotiation.cc:407] Connection header received
0618 22:44:13.231320 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:13.231327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:13.231413 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:13.231482 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:13.239382 (+ 7900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:13.240500 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:13.242316 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:13.243040 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:13.243133 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:13.246161 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:44:13.246168 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:44:13.246172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:44:13.246223 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:44:13.249821 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:13.250620 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:13.250641 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:13.250644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:13.250802 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:44:13.251255 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:44:13.251260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:44:13.251264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:44:13.251629 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:44:13.251856 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:13.252425 (+ 569us) server_negotiation.cc:299] Negotiation successful
0618 22:44:13.252645 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260618 22:44:13.255489 29055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:44:13.264514 28937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fef366de892b4719914e1c618f6ae0c2: No bootstrap required, opened a new log
I20260618 22:44:13.294566 28937 raft_consensus.cc:348] T 00000000000000000000000000000000 P fef366de892b4719914e1c618f6ae0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d60df7d5aa49529da75283f6e93198" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } } peers { permanent_uuid: "fef366de892b4719914e1c618f6ae0c2" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } } peers { permanent_uuid: "920e14c4d60141c89a4376089515c92d" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } }
I20260618 22:44:13.295122 28937 raft_consensus.cc:729] T 00000000000000000000000000000000 P fef366de892b4719914e1c618f6ae0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef366de892b4719914e1c618f6ae0c2, State: Initialized, Role: FOLLOWER
I20260618 22:44:13.295759 28937 consensus_queue.cc:260] T 00000000000000000000000000000000 P fef366de892b4719914e1c618f6ae0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d60df7d5aa49529da75283f6e93198" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } } peers { permanent_uuid: "fef366de892b4719914e1c618f6ae0c2" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } } peers { permanent_uuid: "920e14c4d60141c89a4376089515c92d" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } }
I20260618 22:44:13.297998 29058 sys_catalog.cc:455] T 00000000000000000000000000000000 P fef366de892b4719914e1c618f6ae0c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d60df7d5aa49529da75283f6e93198" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } } peers { permanent_uuid: "fef366de892b4719914e1c618f6ae0c2" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } } peers { permanent_uuid: "920e14c4d60141c89a4376089515c92d" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } } }
I20260618 22:44:13.298600 29058 sys_catalog.cc:458] T 00000000000000000000000000000000 P fef366de892b4719914e1c618f6ae0c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:44:13.299948 28986 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d [term 1 FOLLOWER]: Refusing update from remote peer 37d60df7d5aa49529da75283f6e93198: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:44:13.301470 29045 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [LEADER]: Connected to new peer: Peer: permanent_uuid: "920e14c4d60141c89a4376089515c92d" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:44:13.303345 28937 sys_catalog.cc:564] T 00000000000000000000000000000000 P fef366de892b4719914e1c618f6ae0c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:44:13.303814 28911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fef366de892b4719914e1c618f6ae0c2 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:44:13.307117 28912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "37d60df7d5aa49529da75283f6e93198" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fef366de892b4719914e1c618f6ae0c2"
I20260618 22:44:13.307474 28911 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fef366de892b4719914e1c618f6ae0c2 [term 1 FOLLOWER]: Refusing update from remote peer 37d60df7d5aa49529da75283f6e93198: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:44:13.308367 29032 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef366de892b4719914e1c618f6ae0c2" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:44:13.310781 28912 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fef366de892b4719914e1c618f6ae0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37d60df7d5aa49529da75283f6e93198 in term 1.
I20260618 22:44:13.338393 29058 sys_catalog.cc:455] T 00000000000000000000000000000000 P fef366de892b4719914e1c618f6ae0c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37d60df7d5aa49529da75283f6e93198. Latest consensus state: current_term: 1 leader_uuid: "37d60df7d5aa49529da75283f6e93198" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d60df7d5aa49529da75283f6e93198" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } } peers { permanent_uuid: "fef366de892b4719914e1c618f6ae0c2" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } } peers { permanent_uuid: "920e14c4d60141c89a4376089515c92d" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } } }
I20260618 22:44:13.338775 29058 sys_catalog.cc:458] T 00000000000000000000000000000000 P fef366de892b4719914e1c618f6ae0c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:44:13.365983 29062 mvcc.cc:205] Tried to move back new op lower bound from 7298345587857473536 to 7298345587514576896. Current Snapshot: MvccSnapshot[applied={T|T < 7298345587857473536}]
I20260618 22:44:13.374433 29055 catalog_manager.cc:1347] Generated new cluster ID: 0da47445ec9f4fcabb89d7212810201a
I20260618 22:44:13.374554 29055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:44:13.392128 29045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "37d60df7d5aa49529da75283f6e93198" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d60df7d5aa49529da75283f6e93198" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } } peers { permanent_uuid: "fef366de892b4719914e1c618f6ae0c2" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } } peers { permanent_uuid: "920e14c4d60141c89a4376089515c92d" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } } }
I20260618 22:44:13.392433 29045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [sys.catalog]: This master's current role is: LEADER
I20260618 22:44:13.393082 29058 sys_catalog.cc:455] T 00000000000000000000000000000000 P fef366de892b4719914e1c618f6ae0c2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "37d60df7d5aa49529da75283f6e93198" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d60df7d5aa49529da75283f6e93198" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } } peers { permanent_uuid: "fef366de892b4719914e1c618f6ae0c2" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } } peers { permanent_uuid: "920e14c4d60141c89a4376089515c92d" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } } }
I20260618 22:44:13.393437 29058 sys_catalog.cc:458] T 00000000000000000000000000000000 P fef366de892b4719914e1c618f6ae0c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:44:13.397120 29018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37d60df7d5aa49529da75283f6e93198. Latest consensus state: current_term: 1 leader_uuid: "37d60df7d5aa49529da75283f6e93198" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d60df7d5aa49529da75283f6e93198" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } } peers { permanent_uuid: "fef366de892b4719914e1c618f6ae0c2" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } } peers { permanent_uuid: "920e14c4d60141c89a4376089515c92d" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } } }
I20260618 22:44:13.397394 29018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:44:13.408687 29032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "37d60df7d5aa49529da75283f6e93198" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d60df7d5aa49529da75283f6e93198" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } } peers { permanent_uuid: "fef366de892b4719914e1c618f6ae0c2" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } } peers { permanent_uuid: "920e14c4d60141c89a4376089515c92d" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } } }
I20260618 22:44:13.409511 29032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198 [sys.catalog]: This master's current role is: LEADER
I20260618 22:44:13.430833 29018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "37d60df7d5aa49529da75283f6e93198" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d60df7d5aa49529da75283f6e93198" member_type: VOTER last_known_addr { host: "127.5.213.254" port: 43177 } } peers { permanent_uuid: "fef366de892b4719914e1c618f6ae0c2" member_type: VOTER last_known_addr { host: "127.5.213.253" port: 43849 } } peers { permanent_uuid: "920e14c4d60141c89a4376089515c92d" member_type: VOTER last_known_addr { host: "127.5.213.252" port: 41179 } } }
I20260618 22:44:13.431128 29018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:44:13.464366 29055 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:44:13.466575 29055 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:44:13.481230 29077 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260618 22:44:13.496349 29055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37d60df7d5aa49529da75283f6e93198: Generated new TSK 0
I20260618 22:44:13.497524 29055 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 22:44:13.574095 28941 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:13.560703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58818 (local address 127.5.213.253:43849)
0618 22:44:13.560884 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:13.560890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:13.571675 (+ 10785us) server_negotiation.cc:407] Connection header received
0618 22:44:13.572265 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:13.572274 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:13.572380 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:13.572466 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:13.573797 (+ 1331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58818: BlockingRecv error: recv got EOF from 127.0.0.1:58818 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260618 22:44:13.577219 29016 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:13.560488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59084 (local address 127.5.213.254:43177)
0618 22:44:13.560707 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:13.560714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:13.575219 (+ 14505us) server_negotiation.cc:407] Connection header received
0618 22:44:13.575503 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:13.575510 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:13.575640 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:13.575734 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:13.576897 (+ 1163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59084: BlockingRecv error: recv got EOF from 127.0.0.1:59084 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260618 22:44:13.579938 29014 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:13.567448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37916 (local address 127.5.213.252:41179)
0618 22:44:13.567657 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:13.567664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:13.578057 (+ 10393us) server_negotiation.cc:407] Connection header received
0618 22:44:13.578335 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:13.578342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:13.578436 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:13.578510 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:13.579604 (+ 1094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37916: BlockingRecv error: recv got EOF from 127.0.0.1:37916 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260618 22:44:13.591038 28941 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:13.588781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58832 (local address 127.5.213.253:43849)
0618 22:44:13.589006 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:13.589014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:13.589404 (+ 390us) server_negotiation.cc:407] Connection header received
0618 22:44:13.589650 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:13.589654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:13.589770 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:13.589869 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:13.590728 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58832: BlockingRecv error: recv got EOF from 127.0.0.1:58832 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260618 22:44:13.597606 28941 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:13.595458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58838 (local address 127.5.213.253:43849)
0618 22:44:13.595642 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:13.595649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:13.596206 (+ 557us) server_negotiation.cc:407] Connection header received
0618 22:44:13.596405 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:13.596412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:13.596510 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:13.596632 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:13.597364 (+ 732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58838: BlockingRecv error: recv got EOF from 127.0.0.1:58838 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260618 22:44:13.604135 28941 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:13.602414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58840 (local address 127.5.213.253:43849)
0618 22:44:13.602595 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:13.602601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:13.602910 (+ 309us) server_negotiation.cc:407] Connection header received
0618 22:44:13.603049 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:13.603055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:13.603156 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:13.603271 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:13.603898 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58840: BlockingRecv error: recv got EOF from 127.0.0.1:58840 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260618 22:44:13.612512 28941 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:13.610799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.5.213.253:43849)
0618 22:44:13.610971 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:13.610976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:13.611319 (+ 343us) server_negotiation.cc:407] Connection header received
0618 22:44:13.611465 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:13.611471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:13.611564 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:13.611682 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:13.612299 (+ 617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58850: BlockingRecv error: recv got EOF from 127.0.0.1:58850 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260618 22:44:13.625162 28941 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:13.623437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58854 (local address 127.5.213.253:43849)
0618 22:44:13.623619 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:13.623626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:13.623889 (+ 263us) server_negotiation.cc:407] Connection header received
0618 22:44:13.624049 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:13.624055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:13.624159 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:13.624274 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:13.624928 (+ 654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58854: BlockingRecv error: recv got EOF from 127.0.0.1:58854 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260618 22:44:13.646868 28941 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:13.644862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58856 (local address 127.5.213.253:43849)
0618 22:44:13.645086 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:13.645092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:13.645455 (+ 363us) server_negotiation.cc:407] Connection header received
0618 22:44:13.645650 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:13.645654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:13.645728 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:13.645813 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:13.646614 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58856: BlockingRecv error: recv got EOF from 127.0.0.1:58856 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260618 22:44:13.685189 28941 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:13.683192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58868 (local address 127.5.213.253:43849)
0618 22:44:13.683419 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:13.683425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:13.683736 (+ 311us) server_negotiation.cc:407] Connection header received
0618 22:44:13.683929 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:13.683933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:13.684005 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:13.684087 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:13.684921 (+ 834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58868: BlockingRecv error: recv got EOF from 127.0.0.1:58868 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260618 22:44:13.754822 28941 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:13.752872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.5.213.253:43849)
0618 22:44:13.753062 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:13.753067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:13.753420 (+ 353us) server_negotiation.cc:407] Connection header received
0618 22:44:13.753591 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:13.753595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:13.753682 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:13.753775 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:13.754565 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58880: BlockingRecv error: recv got EOF from 127.0.0.1:58880 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260618 22:44:13.890069 28941 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:13.888122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58886 (local address 127.5.213.253:43849)
0618 22:44:13.888354 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:13.888361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:13.888690 (+ 329us) server_negotiation.cc:407] Connection header received
0618 22:44:13.888870 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:13.888875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:13.888946 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:13.889031 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:13.889809 (+ 778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58886: BlockingRecv error: recv got EOF from 127.0.0.1:58886 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260618 22:44:14.047132 29029 catalog_manager.cc:1259] Loaded cluster ID: 0da47445ec9f4fcabb89d7212810201a
I20260618 22:44:14.047243 29029 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d: loading cluster ID for follower catalog manager: success
I20260618 22:44:14.049890 29029 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d: acquiring CA information for follower catalog manager: success
I20260618 22:44:14.051755 29029 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 920e14c4d60141c89a4376089515c92d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260618 22:44:14.151580 28941 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:44:14.149859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58900 (local address 127.5.213.253:43849)
0618 22:44:14.150039 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:14.150043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:14.150305 (+ 262us) server_negotiation.cc:407] Connection header received
0618 22:44:14.150430 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:14.150434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:14.150528 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:14.150686 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:44:14.151363 (+ 677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58900: BlockingRecv error: recv got EOF from 127.0.0.1:58900 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 22:44:14.485561 29077 catalog_manager.cc:1259] Loaded cluster ID: 0da47445ec9f4fcabb89d7212810201a
I20260618 22:44:14.485678 29077 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fef366de892b4719914e1c618f6ae0c2: loading cluster ID for follower catalog manager: success
I20260618 22:44:14.488407 29077 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fef366de892b4719914e1c618f6ae0c2: acquiring CA information for follower catalog manager: success
I20260618 22:44:14.490449 29077 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fef366de892b4719914e1c618f6ae0c2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 22:44:14.684454 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:14.677819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58916 (local address 127.5.213.253:43849)
0618 22:44:14.678541 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:14.678547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:14.678569 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:44:14.678660 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:14.678664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:14.678811 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:14.678921 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:44:14.679389 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:14.680455 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:14.681877 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:14.682348 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:14.682460 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:14.683017 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:44:14.683623 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:44:14.683787 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:14.683892 (+ 105us) server_negotiation.cc:299] Negotiation successful
0618 22:44:14.683984 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":434,"threads_started":1}
I20260618 22:44:14.700736 29150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:14.690783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37920 (local address 127.5.213.252:41179)
0618 22:44:14.695022 (+ 4239us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:14.695026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:14.695048 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:44:14.695159 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:14.695163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:14.695325 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:14.695531 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:44:14.696028 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:14.696826 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:14.698280 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:14.698785 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:14.698912 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:14.699510 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:44:14.700044 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:44:14.700194 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:14.700329 (+ 135us) server_negotiation.cc:299] Negotiation successful
0618 22:44:14.700432 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4125,"thread_start_us":4023,"threads_started":1}
I20260618 22:44:14.727461 29158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:14.708281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59100 (local address 127.5.213.254:43177)
0618 22:44:14.719369 (+ 11088us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:14.719378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:14.719404 (+ 26us) server_negotiation.cc:407] Connection header received
0618 22:44:14.719551 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:14.719558 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:14.720015 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:14.720248 (+ 233us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:44:14.721159 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:14.722371 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:14.724505 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:14.725118 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:14.725774 (+ 656us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:14.725984 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:44:14.726434 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:44:14.727007 (+ 573us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:14.727056 (+ 49us) server_negotiation.cc:299] Negotiation successful
0618 22:44:14.727139 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10927,"thread_start_us":11533,"threads_started":1}
I20260618 22:44:14.727622 29150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:14.708539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37930 (local address 127.5.213.252:41179)
0618 22:44:14.708721 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:14.708726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:14.720996 (+ 12270us) server_negotiation.cc:407] Connection header received
0618 22:44:14.721137 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:14.721143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:14.721239 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:14.721349 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:44:14.722148 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:14.723220 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:14.725477 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:14.726121 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:14.726220 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:14.726868 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:44:14.726992 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:44:14.727054 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:14.727263 (+ 209us) server_negotiation.cc:299] Negotiation successful
0618 22:44:14.727345 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 22:44:14.728901 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:44:14.707776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58930 (local address 127.5.213.253:43849)
0618 22:44:14.708067 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0618 22:44:14.708076 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:44:14.721583 (+ 13507us) server_negotiation.cc:407] Connection header received
0618 22:44:14.721911 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:44:14.721917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:44:14.722005 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:44:14.722062 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:44:14.722785 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:14.723887 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:14.726306 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:44:14.726953 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:44:14.727049 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:44:14.727771 (+ 722us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:44:14.727895 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:44:14.727959 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 22:44:14.728498 (+ 539us) server_negotiation.cc:299] Negotiation successful
0618 22:44:14.728572 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260618 22:44:14.732769 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 28796
I20260618 22:44:14.734038 28858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:44:14.868857 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 28796
I20260618 22:44:14.885479 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 28870
I20260618 22:44:14.886541 28932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:44:14.992434 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 28870
I20260618 22:44:15.008116 5975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu pid 28944
I20260618 22:44:15.009174 29006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260618 22:44:15.114996 5975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnPxcP/build/asan/bin/kudu with pid 28944
2026-06-18T22:44:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4818 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9221 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (347246 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST